Nov 02 17:26:52.822105 ip-10-0-141-87 systemd[1]: kubelet.service: Failed to load environment files: No such file or directory
Nov 02 17:26:52.822116 ip-10-0-141-87 systemd[1]: kubelet.service: Failed to run 'start-pre' task: No such file or directory
Nov 02 17:26:52.822125 ip-10-0-141-87 systemd[1]: kubelet.service: Failed with result 'resources'.
Nov 02 17:26:52.822457 ip-10-0-141-87 systemd[1]: Failed to start Kubernetes Kubelet.
Nov 02 17:27:03.007054 ip-10-0-141-87 systemd[1]: kubelet.service: Failed to schedule restart job: Unit crio.service not found.
Nov 02 17:27:03.007070 ip-10-0-141-87 systemd[1]: kubelet.service: Failed with result 'resources'.
-- Boot c30eb82f014143199151e90a0e6750c5 --
Nov 02 17:29:02.013386 ip-10-0-141-87 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 17:29:02.400656 ip-10-0-141-87 kubenswrapper[2542]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:29:02.400656 ip-10-0-141-87 kubenswrapper[2542]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 17:29:02.400656 ip-10-0-141-87 kubenswrapper[2542]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:29:02.400656 ip-10-0-141-87 kubenswrapper[2542]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:29:02.406307 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406219    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:02.406307 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406306    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:02.406307 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406311    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406314    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406317    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406320    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406323    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406325    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406328    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406330    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406333    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406335    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406338    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406340    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406343    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406345    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406348    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406351    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406353    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406356    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406358    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406366    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:02.406437 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406369    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406372    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406375    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406377    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406379    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406382    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406384    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406387    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406389    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406392    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406394    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406396    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406399    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406401    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406406    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406409    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406412    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406414    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406417    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:02.406908 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406419    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406422    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406425    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406428    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406430    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406434    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406438    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406442    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406445    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406447    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406451    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406454    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406456    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406459    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406462    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406466    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406468    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406471    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406473    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:02.407353 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406476    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406478    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406480    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406483    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406485    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406488    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406491    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406494    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406498    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406503    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406505    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406508    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406511    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406514    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406517    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406519    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406522    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406524    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406527    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:02.407802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406529    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406531    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406534    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406536    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406539    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406541    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406544    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406546    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406549    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406551    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406554    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406558    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406560    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406563    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406565    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406567    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406570    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406581    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406585    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:02.408262 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406587    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406590    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406593    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406595    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406598    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406600    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406603    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.406606    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407398    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407405    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407408    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407412    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407417    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407420    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407423    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407426    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407429    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407431    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407434    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407437    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:02.408766 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407440    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407442    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407445    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407447    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407450    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407453    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407455    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407458    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407460    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407462    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407465    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407468    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407470    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407473    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407476    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407479    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407481    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407484    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407487    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407489    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:02.409277 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407492    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407495    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407497    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407500    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407503    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407505    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407508    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407512    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407515    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407518    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407520    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407523    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407525    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407527    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407530    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407532    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407535    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407537    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407540    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:02.409788 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407542    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407544    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407547    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407549    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407553    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407555    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407558    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407561    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407563    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407566    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407568    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407571    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407573    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407576    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407580    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407582    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407585    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407587    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407589    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407592    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:02.410249 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407594    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407597    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407599    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407602    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407604    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407607    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407609    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407612    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407615    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407618    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407620    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407623    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407625    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407628    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407630    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407633    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407635    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407638    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407640    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407643    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:02.410722 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407646    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407648    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407651    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407653    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407656    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407658    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407661    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407664    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407667    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407669    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407672    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407674    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407677    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407679    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.407681    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408213    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408220    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408224    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408227    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:02.411248 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408230    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408233    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408236    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408239    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408241    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408244    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408247    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408249    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408252    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408255    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408258    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408261    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408263    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408266    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408269    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408271    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408274    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408277    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408280    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408282    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:02.411691 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408286    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408288    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408291    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408293    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408296    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408298    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408301    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408303    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408306    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408308    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408311    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408313    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408317    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408319    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408322    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408325    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408328    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408330    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408332    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:02.412172 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408335    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408337    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408340    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408342    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408345    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408347    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408350    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408352    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408354    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408357    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408359    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408361    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408364    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408366    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408371    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408374    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408378    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408381    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408383    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:02.412622 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408386    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408389    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408391    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408394    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408396    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408399    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408404    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408407    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408409    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408412    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408414    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408417    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408420    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408422    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408425    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408428    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408430    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408435    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408437    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408440    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:02.413066 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408442    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408444    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408447    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408450    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408452    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408455    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408457    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408460    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408466    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408469    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408471    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408474    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408476    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408479    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408481    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408483    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408486    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408489    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408493    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408495    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:02.413544 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408498    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408502    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408506    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.408508    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409443    2542 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409453    2542 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409459    2542 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409464    2542 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409470    2542 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409476    2542 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409480    2542 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409484    2542 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409487    2542 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409490    2542 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409494    2542 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409497    2542 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409501    2542 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409504    2542 flags.go:64] FLAG: --cgroup-root=""
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409506    2542 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409510    2542 flags.go:64] FLAG: --client-ca-file=""
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409512    2542 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409516    2542 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409521    2542 flags.go:64] FLAG: --cluster-domain=""
Nov 02 17:29:02.414019 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409524    2542 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409527    2542 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409530    2542 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409533    2542 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409537    2542 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409540    2542 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409543    2542 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409547    2542 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409551    2542 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409553    2542 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409556    2542 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409560    2542 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409563    2542 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409567    2542 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409570    2542 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409573    2542 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409576    2542 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409579    2542 flags.go:64] FLAG: --enable-server="true"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409583    2542 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409587    2542 flags.go:64] FLAG: --event-burst="100"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409590    2542 flags.go:64] FLAG: --event-qps="50"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409593    2542 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409596    2542 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409599    2542 flags.go:64] FLAG: --eviction-hard=""
Nov 02 17:29:02.414557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409604    2542 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409607    2542 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409610    2542 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409613    2542 flags.go:64] FLAG: --eviction-soft=""
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409616    2542 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409619    2542 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409622    2542 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409625    2542 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409628    2542 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409631    2542 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409634    2542 flags.go:64] FLAG: --feature-gates=""
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409638    2542 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409641    2542 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409644    2542 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409647    2542 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409650    2542 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409654    2542 flags.go:64] FLAG: --help="false"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409657    2542 flags.go:64] FLAG: --hostname-override="ip-10-0-141-87.ec2.internal"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409660    2542 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409663    2542 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409666    2542 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409669    2542 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409672    2542 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409675    2542 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 17:29:02.415235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409678    2542 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409681    2542 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409686    2542 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409689    2542 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409692    2542 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409695    2542 flags.go:64] FLAG: --kube-reserved=""
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409698    2542 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409701    2542 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409704    2542 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409707    2542 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409710    2542 flags.go:64] FLAG: --lock-file=""
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409712    2542 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409715    2542 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409718    2542 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409724    2542 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409727    2542 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409730    2542 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409733    2542 flags.go:64] FLAG: --logging-format="text"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409736    2542 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409740    2542 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409742    2542 flags.go:64] FLAG: --manifest-url=""
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409745    2542 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409750    2542 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409753    2542 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409757    2542 flags.go:64] FLAG: --max-pods="110"
Nov 02 17:29:02.415816 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409761    2542 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409764    2542 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409767    2542 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409770    2542 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409773    2542 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409776    2542 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409779    2542 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409787    2542 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409791    2542 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409796    2542 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409799    2542 flags.go:64] FLAG: --pod-cidr=""
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409802    2542 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409805    2542 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409808    2542 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409811    2542 flags.go:64] FLAG: --port="10250"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409814    2542 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409817    2542 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-086bc04b1e25d38e7"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409820    2542 flags.go:64] FLAG: --qos-reserved=""
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409823    2542 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409825    2542 flags.go:64] FLAG: --register-node="true"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409828    2542 flags.go:64] FLAG: --register-with-taints=""
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409833    2542 flags.go:64] FLAG: --registry-burst="10"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409836    2542 flags.go:64] FLAG: --registry-qps="5"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409839    2542 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409842    2542 flags.go:64] FLAG: --reserved-memory=""
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409846    2542 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 17:29:02.416439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409849    2542 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409852    2542 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409854    2542 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409857    2542 flags.go:64] FLAG: --runonce="false"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409860    2542 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409863    2542 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409866    2542 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409869    2542 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409872    2542 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409875    2542 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409878    2542 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409882    2542 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409884    2542 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409887    2542 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409890    2542 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409894    2542 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409897    2542 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409900    2542 flags.go:64] FLAG: --system-cgroups=""
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409903    2542 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409909    2542 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409912    2542 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409914    2542 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409919    2542 flags.go:64] FLAG: --tls-min-version=""
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409922    2542 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409925    2542 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 17:29:02.417051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409928    2542 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409931    2542 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409934    2542 flags.go:64] FLAG: --v="2"
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409939    2542 flags.go:64] FLAG: --version="false"
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409943    2542 flags.go:64] FLAG: --vmodule=""
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409947    2542 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.409951    2542 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410093    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410112    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410116    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410119    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410122    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410125    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410128    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410131    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410133    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410136    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410138    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410141    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410143    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410146    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:02.417646 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410149    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410154    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410159    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410162    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410165    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410167    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410170    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410173    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410175    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410178    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410180    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410182    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410185    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410188    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410190    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410193    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410195    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410198    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410201    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:02.418170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410205    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410207    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410210    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410212    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410215    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410217    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410220    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410222    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410224    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410227    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410229    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410232    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410234    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410236    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410240    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410243    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410245    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410248    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410251    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410253    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:02.418640 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410256    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410258    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410261    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410264    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410266    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410269    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410271    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410274    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410276    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410279    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410281    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410284    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410289    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410291    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410294    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410296    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410299    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410301    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410304    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410306    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:02.419113 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410308    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410311    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410313    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410316    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410318    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410321    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410324    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410326    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410329    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410331    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410334    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410336    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410338    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410341    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410343    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410345    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410348    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410350    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410353    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:02.419589 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410355    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410358    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410360    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410362    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410365    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410369    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410372    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410374    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410377    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410379    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410382    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410384    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410387    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410389    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.410417    2542 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410451    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:02.420071 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410457    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410461    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410464    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410467    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410486    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410490    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410494    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410497    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410500    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410502    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410505    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410507    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410509    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410512    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410515    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410517    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410519    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410522    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410524    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:02.420475 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410527    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410529    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410533    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410535    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410539    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410541    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410544    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410546    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410549    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410551    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410553    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410556    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410558    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410561    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410563    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410565    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410568    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410580    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410583    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410586    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:02.420969 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410588    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410591    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410593    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410596    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410598    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410600    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410603    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410606    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410608    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410610    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410613    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410615    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410618    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410620    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410624    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410626    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410629    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410632    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410634    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410637    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:02.421463 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410639    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410642    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410646    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410649    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410651    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410653    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410656    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410658    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410661    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410663    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410666    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410668    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410670    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410673    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410675    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410677    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410680    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410682    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410685    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:02.421946 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410687    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410689    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410692    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410694    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410697    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410699    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410701    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410705    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410707    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410710    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410712    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410718    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410721    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410725    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410728    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410731    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410734    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410737    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410740    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:02.422401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410743    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410745    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410749    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410752    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410754    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410757    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410759    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.410763    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.417459    2542 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.417475    2542 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.417492    2542 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.417497    2542 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417554    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417559    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417563    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417568    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417571    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417575    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417578    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:02.422863 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417580    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417583    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417586    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417589    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417591    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417594    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417596    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417599    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417602    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417604    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417607    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417609    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417612    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417614    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417617    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417619    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417622    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417625    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417627    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:02.423472 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417630    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417632    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417635    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417638    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417641    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417644    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417646    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417649    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417652    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417654    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417657    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417660    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417663    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417665    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417668    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417670    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417673    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417675    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417678    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417680    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:02.423965 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417683    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417685    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417688    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417690    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417693    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417695    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417698    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417700    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417702    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417705    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417708    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417711    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417713    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417716    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417718    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417721    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417725    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417727    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417730    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:02.424467 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417732    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417734    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417737    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417739    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417742    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417744    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417747    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417749    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417753    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417756    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417759    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417761    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417764    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417767    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417769    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417772    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417774    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417777    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417779    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417782    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:02.425221 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417785    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417787    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417790    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417793    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417795    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417798    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417800    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417803    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417805    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417808    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417811    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417813    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417816    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417818    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417825    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417828    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417831    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417834    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417836    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417839    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:02.425873 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.417841    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.417869    2542 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418009    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418013    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418016    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418019    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418021    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418024    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418027    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418029    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418032    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418034    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418036    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418039    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418041    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418044    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:02.426399 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418046    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418048    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418051    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418053    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418055    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418058    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418060    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418065    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418068    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418070    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418072    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418075    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418078    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418080    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418083    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418085    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418087    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418090    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418092    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418094    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:02.426789 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418097    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418099    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418121    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418124    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418126    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418129    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418132    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418134    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418136    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418139    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418141    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418144    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418146    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418148    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418151    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418154    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418156    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418158    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418161    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:02.427318 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418163    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418166    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418169    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418171    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418174    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418177    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418180    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418182    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418185    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418187    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418190    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418192    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418195    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418197    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418200    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418202    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418205    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418207    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418210    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:02.427933 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418212    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418215    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418217    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418220    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418222    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418224    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418227    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418229    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418231    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418235    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418239    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418242    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418244    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418247    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418249    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418252    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418255    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418257    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418259    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:02.428419 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418262    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418264    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418267    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418269    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418271    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418274    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418276    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418278    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418281    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418283    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418285    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418288    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418290    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418294    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418298    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.418322    2542 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:29:02.429092 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418354    2542 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418357    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418360    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418363    2542 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418366    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418368    2542 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418371    2542 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418374    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418376    2542 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418378    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418381    2542 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418383    2542 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418385    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418388    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418391    2542 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418393    2542 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418396    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418398    2542 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418401    2542 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418403    2542 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:02.429518 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418405    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418408    2542 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418411    2542 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418413    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418415    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418418    2542 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418420    2542 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418423    2542 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418425    2542 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418428    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418430    2542 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418433    2542 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418435    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418438    2542 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418440    2542 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418442    2542 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418444    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418447    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418450    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418452    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:02.429983 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418455    2542 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418457    2542 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418459    2542 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418462    2542 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418464    2542 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418467    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418469    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418472    2542 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418474    2542 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418477    2542 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418479    2542 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418482    2542 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418484    2542 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418486    2542 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418489    2542 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418492    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418494    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418496    2542 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418498    2542 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:02.430543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418501    2542 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418506    2542 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418508    2542 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418511    2542 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418513    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418516    2542 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418518    2542 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418521    2542 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418523    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418526    2542 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418528    2542 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418531    2542 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418533    2542 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418536    2542 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418538    2542 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418541    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418544    2542 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418548    2542 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418551    2542 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:02.431026 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418553    2542 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418556    2542 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418559    2542 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418561    2542 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418563    2542 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418566    2542 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418568    2542 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418578    2542 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418581    2542 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418583    2542 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418586    2542 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418588    2542 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418591    2542 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418593    2542 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418597    2542 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418600    2542 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418602    2542 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418605    2542 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418607    2542 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:02.431504 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418610    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:02.431943 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418614    2542 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:02.431943 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418618    2542 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:02.431943 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418621    2542 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:02.431943 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418624    2542 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:02.431943 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418626    2542 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:02.431943 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418629    2542 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:02.431943 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418631    2542 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:02.431943 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:02.418634    2542 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:02.431943 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.418685    2542 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 17:29:02.431943 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.420620    2542 bootstrap.go:101] "Use the bootstrap credentials to request a cert, and set kubeconfig to point to the certificate dir"
Nov 02 17:29:02.431943 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.421571    2542 server.go:1014] "Starting client certificate rotation"
Nov 02 17:29:02.431943 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.421665    2542 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 17:29:02.431943 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.421698    2542 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 17:29:02.431943 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.423541    2542 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:29:02.431943 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.430416    2542 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:29:02.440044 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.440028    2542 log.go:25] "Validated CRI v1 runtime API"
Nov 02 17:29:02.443568 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.443545    2542 log.go:25] "Validated CRI v1 image API"
Nov 02 17:29:02.444979 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.444952    2542 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 17:29:02.447766 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.447747    2542 fs.go:135] Filesystem UUIDs: map[2577969f-b148-4472-84ae-e5ed82a01786:/dev/nvme0n1p3 6ee8195b-f86e-4e62-94df-29f84cdf6339:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 02 17:29:02.447848 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.447764    2542 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0}]
Nov 02 17:29:02.452886 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.452775    2542 manager.go:217] Machine: {Timestamp:2026-11-02 17:29:02.45141232 +0000 UTC m=+0.321785085 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100129 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eefb5bbdc0967233fbc8f25552fee SystemUUID:ec2eefb5-bbdc-0967-233f-bc8f25552fee BootID:c30eb82f-0141-4319-9151-e90a0e6750c5 Filesystems:[{Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:br-ex MacAddress:02:ff:ce:93:b2:55 Speed:0 Mtu:9001} {Name:ens5 MacAddress:02:ff:ce:93:b2:55 Speed:0 Mtu:9001} {Name:ovs-system MacAddress:ce:07:77:91:5b:88 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 17:29:02.452886 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.452884    2542 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 17:29:02.452997 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.452959    2542 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 17:29:02.455836 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.455807    2542 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 17:29:02.455998 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.455837    2542 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-87.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 17:29:02.456085 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.456004    2542 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 17:29:02.456085 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.456016    2542 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 17:29:02.456196 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.456088    2542 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:29:02.456196 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.456138    2542 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:29:02.456990 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.456976    2542 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 17:29:02.460748 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.460733    2542 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 17:29:02.460880 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.460869    2542 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 17:29:02.461586 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.461574    2542 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 17:29:02.461642 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.461592    2542 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 17:29:02.461642 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.461607    2542 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 17:29:02.461642 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.461619    2542 kubelet.go:405] "Adding apiserver pod source"
Nov 02 17:29:02.461642 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.461632    2542 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 17:29:02.462014 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.462000    2542 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:29:02.464176 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.464152    2542 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 17:29:02.468134 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.468121    2542 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:29:02.469059 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.469044    2542 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 17:29:02.469144 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.469090    2542 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 17:29:02.469750 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.469735    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 17:29:02.469787 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.469762    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 17:29:02.469787 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.469775    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 17:29:02.469837 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.469791    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 17:29:02.469837 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.469800    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 17:29:02.469837 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.469807    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 17:29:02.470260 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.470251    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 17:29:02.470290 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.470261    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 17:29:02.470290 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.470267    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 17:29:02.470290 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.470287    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 17:29:02.470364 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.470300    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 17:29:02.470364 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.470309    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 17:29:02.471420 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.471409    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 17:29:02.471452 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.471423    2542 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 17:29:02.475337 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.475322    2542 server.go:1263] "Started kubelet"
Nov 02 17:29:02.475425 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.475359    2542 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 17:29:02.475991 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.475937    2542 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 17:29:02.476079 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.476019    2542 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 17:29:02.476293 ip-10-0-141-87 systemd[1]: Started Kubernetes Kubelet.
Nov 02 17:29:02.479243 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.479219    2542 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 17:29:02.482311 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.482293    2542 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 17:29:02.482388 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:02.482350    2542 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="failed to get imageFs info: unable to find data in memory cache"
Nov 02 17:29:02.484314 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.484298    2542 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 17:29:02.485198 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.485179    2542 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 17:29:02.485290 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:02.485271    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:02.485720 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.485704    2542 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 17:29:02.485807 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.485730    2542 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 17:29:02.485807 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.485739    2542 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 17:29:02.485894 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.485877    2542 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 17:29:02.485894 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.485885    2542 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 17:29:02.485988 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.485982    2542 factory.go:55] Registering systemd factory
Nov 02 17:29:02.486221 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.485993    2542 factory.go:223] Registration of the systemd container factory successfully
Nov 02 17:29:02.486221 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.486204    2542 factory.go:153] Registering CRI-O factory
Nov 02 17:29:02.486221 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.486215    2542 factory.go:223] Registration of the crio container factory successfully
Nov 02 17:29:02.486360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.486251    2542 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 17:29:02.486360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.486264    2542 factory.go:103] Registering Raw factory
Nov 02 17:29:02.486360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.486276    2542 manager.go:1199] Started watching for new ooms in manager
Nov 02 17:29:02.487130 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.487098    2542 manager.go:319] Starting recovery of all containers
Nov 02 17:29:02.496956 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.496935    2542 manager.go:324] Recovery completed
Nov 02 17:29:02.501010 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.500997    2542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:29:02.502840 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.502815    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 17:29:02.502900 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.502849    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 17:29:02.502900 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.502859    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID"
Nov 02 17:29:02.503810 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.503794    2542 cpu_manager.go:226] "Starting" policy="none"
Nov 02 17:29:02.503900 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.503808    2542 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 17:29:02.503900 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.503822    2542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:29:02.503900 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.503836    2542 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 17:29:02.504640 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.504623    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 17:29:02.504703 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.504652    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 17:29:02.504703 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.504664    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID"
Nov 02 17:29:02.505977 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.505962    2542 policy_none.go:50] "Start"
Nov 02 17:29:02.506011 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.505985    2542 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 17:29:02.506011 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.506006    2542 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 17:29:02.508015 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.508002    2542 policy_none.go:44] "Start"
Nov 02 17:29:02.518635 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.518552    2542 csi_plugin.go:990] Failed to contact API server when waiting for CSINode publishing: csinodes.storage.k8s.io "ip-10-0-141-87.ec2.internal" is forbidden: User "system:anonymous" cannot get resource "csinodes" in API group "storage.k8s.io" at the cluster scope
Nov 02 17:29:02.520587 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:02.518554    2542 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-87.ec2.internal.18e44a91fcf79197  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-87.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:Starting,Message:Starting kubelet.,Source:EventSource{Component:kubelet,Host:ip-10-0-141-87.ec2.internal,},FirstTimestamp:2026-11-02 17:29:02.475096471 +0000 UTC m=+0.345469235,LastTimestamp:2026-11-02 17:29:02.475096471 +0000 UTC m=+0.345469235,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-87.ec2.internal,}"
Nov 02 17:29:02.529367 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:02.529263    2542 controller.go:201] "Failed to ensure lease exists, will retry" err="leases.coordination.k8s.io \"ip-10-0-141-87.ec2.internal\" is forbidden: User \"system:anonymous\" cannot get resource \"leases\" in API group \"coordination.k8s.io\" in the namespace \"kube-node-lease\"" interval="200ms"
Nov 02 17:29:02.539255 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:02.539187    2542 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-87.ec2.internal.18e44a91fe9ecff9  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-87.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasSufficientMemory,Message:Node ip-10-0-141-87.ec2.internal status is now: NodeHasSufficientMemory,Source:EventSource{Component:kubelet,Host:ip-10-0-141-87.ec2.internal,},FirstTimestamp:2026-11-02 17:29:02.502834169 +0000 UTC m=+0.373206934,LastTimestamp:2026-11-02 17:29:02.502834169 +0000 UTC m=+0.373206934,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-87.ec2.internal,}"
Nov 02 17:29:02.539395 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:02.539374    2542 reflector.go:204] "Failed to watch" err="failed to list *v1.CSIDriver: csidrivers.storage.k8s.io is forbidden: User \"system:anonymous\" cannot list resource \"csidrivers\" in API group \"storage.k8s.io\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver"
Nov 02 17:29:02.542715 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.542680    2542 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 17:29:02.546868 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.546853    2542 manager.go:341] "Starting Device Plugin manager"
Nov 02 17:29:02.546941 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:02.546884    2542 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 17:29:02.546941 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.546896    2542 server.go:87] "Starting device plugin registration server"
Nov 02 17:29:02.547175 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.547162    2542 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 17:29:02.547233 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.547177    2542 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 17:29:02.547302 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.547278    2542 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 17:29:02.547375 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.547360    2542 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 17:29:02.547375 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.547373    2542 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 17:29:02.547870 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:02.547851    2542 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 17:29:02.547941 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:02.547900    2542 eviction_manager.go:297] "Eviction manager: failed to get summary stats" err="failed to get node info: node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:02.555589 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.555574    2542 reflector.go:446] "Caches populated" logger="kubernetes.io/kube-apiserver-client-kubelet" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 17:29:02.555666 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.555625    2542 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nwvzr"
Nov 02 17:29:02.558802 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.558783    2542 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kube-apiserver-client-kubelet" csr="csr-nwvzr"
Nov 02 17:29:02.586857 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:02.571409    2542 event.go:359] "Server rejected event (will not retry!)" err="events is forbidden: User \"system:anonymous\" cannot create resource \"events\" in API group \"\" in the namespace \"default\"" event="&Event{ObjectMeta:{ip-10-0-141-87.ec2.internal.18e44a91fe9f1b15  default    0 0001-01-01 00:00:00 +0000 UTC <nil> <nil> map[] map[] [] [] []},InvolvedObject:ObjectReference{Kind:Node,Namespace:,Name:ip-10-0-141-87.ec2.internal,UID:,APIVersion:v1,ResourceVersion:,FieldPath:,},Reason:NodeHasNoDiskPressure,Message:Node ip-10-0-141-87.ec2.internal status is now: NodeHasNoDiskPressure,Source:EventSource{Component:kubelet,Host:ip-10-0-141-87.ec2.internal,},FirstTimestamp:2026-11-02 17:29:02.502853397 +0000 UTC m=+0.373226161,LastTimestamp:2026-11-02 17:29:02.502853397 +0000 UTC m=+0.373226161,Count:1,Type:Normal,EventTime:0001-01-01 00:00:00 +0000 UTC,Series:nil,Action:,Related:nil,ReportingController:kubelet,ReportingInstance:ip-10-0-141-87.ec2.internal,}"
Nov 02 17:29:02.586857 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:02.571729    2542 reflector.go:204] "Failed to watch" err="failed to list *v1.Node: nodes \"ip-10-0-141-87.ec2.internal\" is forbidden: User \"system:anonymous\" cannot list resource \"nodes\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node"
Nov 02 17:29:02.586857 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:02.571985    2542 reflector.go:204] "Failed to watch" err="failed to list *v1.Service: services is forbidden: User \"system:anonymous\" cannot list resource \"services\" in API group \"\" at the cluster scope" logger="UnhandledError" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service"
Nov 02 17:29:02.616344 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.616308    2542 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 17:29:02.617662 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.617648    2542 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 17:29:02.617742 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.617669    2542 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 17:29:02.617742 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.617688    2542 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 17:29:02.617742 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:02.617726    2542 kubelet.go:2545] "Skipping pod synchronization" err="PLEG is not healthy: pleg has yet to be successful"
Nov 02 17:29:02.637758 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.637736    2542 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:29:02.647825 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.647810    2542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:29:02.648814 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.648800    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 17:29:02.648862 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.648829    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 17:29:02.648862 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.648838    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID"
Nov 02 17:29:02.648862 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:02.648859    2542 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-87.ec2.internal"
Nov 02 17:29:03.777499 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.777480    2542 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-87.ec2.internal"
Nov 02 17:29:03.777557 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:03.777502    2542 kubelet_node_status.go:524] "Error updating node status, will retry" err="error getting node \"ip-10-0-141-87.ec2.internal\": node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:03.813866 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.813840    2542 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"]
Nov 02 17:29:03.813964 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.813893    2542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:29:03.814800 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.814784    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 17:29:03.814911 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.814824    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 17:29:03.814911 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.814835    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID"
Nov 02 17:29:03.816701 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.816690    2542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:29:03.816868 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.816852    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:03.816930 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.816891    2542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:29:03.818305 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.818278    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 17:29:03.818391 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.818311    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 17:29:03.818391 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.818324    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID"
Nov 02 17:29:03.818391 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.818278    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 17:29:03.818391 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.818383    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 17:29:03.818509 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.818397    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID"
Nov 02 17:29:03.820299 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.820282    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:03.820387 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.820315    2542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:29:03.821607 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.821593    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 17:29:03.821663 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.821617    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 17:29:03.821663 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.821632    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID"
Nov 02 17:29:03.849907 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:03.849888    2542 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-87.ec2.internal\" not found" node="ip-10-0-141-87.ec2.internal"
Nov 02 17:29:03.852750 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:03.852733    2542 kubelet.go:3364] "No need to create a mirror pod, since failed to get node info from the cluster" err="node \"ip-10-0-141-87.ec2.internal\" not found" node="ip-10-0-141-87.ec2.internal"
Nov 02 17:29:03.853255 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:03.853240    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:03.882640 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.882616    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbfcd267e5bdda32002f5b72a55d4954-config\") pod \"kube-apiserver-proxy-ip-10-0-141-87.ec2.internal\" (UID: \"fbfcd267e5bdda32002f5b72a55d4954\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:03.882737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.882645    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/375359043409c490a8ed0d21b8afeaf5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"375359043409c490a8ed0d21b8afeaf5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:03.882737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.882670    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/375359043409c490a8ed0d21b8afeaf5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"375359043409c490a8ed0d21b8afeaf5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:03.954238 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:03.954215    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:03.983562 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.983538    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/375359043409c490a8ed0d21b8afeaf5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"375359043409c490a8ed0d21b8afeaf5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:03.983628 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.983573    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/375359043409c490a8ed0d21b8afeaf5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"375359043409c490a8ed0d21b8afeaf5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:03.983628 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.983590    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbfcd267e5bdda32002f5b72a55d4954-config\") pod \"kube-apiserver-proxy-ip-10-0-141-87.ec2.internal\" (UID: \"fbfcd267e5bdda32002f5b72a55d4954\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:03.983698 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.983634    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbfcd267e5bdda32002f5b72a55d4954-config\") pod \"kube-apiserver-proxy-ip-10-0-141-87.ec2.internal\" (UID: \"fbfcd267e5bdda32002f5b72a55d4954\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:03.983698 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.983654    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/375359043409c490a8ed0d21b8afeaf5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"375359043409c490a8ed0d21b8afeaf5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:03.983698 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:03.983654    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/375359043409c490a8ed0d21b8afeaf5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"375359043409c490a8ed0d21b8afeaf5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:04.055014 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:04.054956    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:04.152789 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:04.152767    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:04.155899 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:04.155879    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:04.156001 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:04.155924    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:04.256549 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:04.256523    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:04.356659 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:04.356580    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:04.457052 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:04.457019    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:04.515956 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:04.515933    2542 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:29:04.517974 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:04.517959    2542 transport.go:147] "Certificate rotation detected, shutting down client connections to start using new credentials"
Nov 02 17:29:04.518073 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:04.518058    2542 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 17:29:04.518118 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:04.518100    2542 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="very short watch: k8s.io/client-go/informers/factory.go:161: Unexpected watch close - watch lasted less than a second and no items received"
Nov 02 17:29:04.557238 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:04.557214    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:04.657614 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:04.657548    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:04.658662 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:04.658612    2542 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:24:03 +0000 UTC" deadline="2028-08-16 01:33:22.139222726 +0000 UTC"
Nov 02 17:29:04.658712 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:04.658664    2542 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="15656h4m17.480562448s"
Nov 02 17:29:04.727191 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:04.727160    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfbfcd267e5bdda32002f5b72a55d4954.slice/crio-ee9534b936fc2c51176c28d0ce4ea47c7e9ff43500af65957a17de521caccfe0 WatchSource:0}: Error finding container ee9534b936fc2c51176c28d0ce4ea47c7e9ff43500af65957a17de521caccfe0: Status 404 returned error can't find the container with id ee9534b936fc2c51176c28d0ce4ea47c7e9ff43500af65957a17de521caccfe0
Nov 02 17:29:04.727543 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:04.727524    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod375359043409c490a8ed0d21b8afeaf5.slice/crio-28fcb0717813a29dc1866541bc3dbb0f0258922adc7924ce1ef6a82d0c101225 WatchSource:0}: Error finding container 28fcb0717813a29dc1866541bc3dbb0f0258922adc7924ce1ef6a82d0c101225: Status 404 returned error can't find the container with id 28fcb0717813a29dc1866541bc3dbb0f0258922adc7924ce1ef6a82d0c101225
Nov 02 17:29:04.729694 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:04.729681    2542 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:29:04.730769 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:04.730750    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 17:29:04.730867 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:04.730783    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 17:29:04.730867 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:04.730795    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID"
Nov 02 17:29:04.733245 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:04.733231    2542 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:29:04.758607 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:04.758571    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:04.859437 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:04.859394    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:04.913208 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:04.913138    2542 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:29:04.960187 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:04.960163    2542 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:04.989527 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:04.989505    2542 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:29:05.081892 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.081865    2542 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:05.108399 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.108377    2542 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:29:05.109541 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.109519    2542 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:05.132289 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.132267    2542 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:29:05.469256 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.469225    2542 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:29:05.558838 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.558797    2542 apiserver.go:52] "Watching apiserver"
Nov 02 17:29:05.581172 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.581144    2542 certificate_manager.go:566] "Rotating certificates" logger="kubernetes.io/kubelet-serving"
Nov 02 17:29:05.589641 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.588524    2542 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 17:29:05.589641 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.589232    2542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-multus/network-metrics-daemon-tk8ht","kube-system/konnectivity-agent-mxsjq","openshift-image-registry/node-ca-lhwgm","openshift-ovn-kubernetes/ovnkube-node-44l94","openshift-multus/multus-7qzkz","openshift-network-operator/iptables-alerter-dlp6w","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n","openshift-cluster-node-tuning-operator/tuned-n9fk9","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal","openshift-network-diagnostics/network-check-target-nttxj","kube-system/global-pull-secret-syncer-mmhd7","kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal","openshift-dns/node-resolver-hhsrb","openshift-multus/multus-additional-cni-plugins-ngkbz"]
Nov 02 17:29:05.592084 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.592062    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:05.592206 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:05.592141    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d"
Nov 02 17:29:05.594243 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.594170    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:05.596614 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.596593    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:05.599107 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.599077    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.601081 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.601060    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.603396 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.603361    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:05.606427 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.606411    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.608710 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.608690    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:05.611343 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.611324    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 17:29:05.611925 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.611908    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.612674 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.612654    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9lngf\""
Nov 02 17:29:05.612765 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.612720    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 17:29:05.614089 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.614069    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 17:29:05.614255 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.614229    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:05.614335 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:05.614315    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8ht" podUID="dd733778-7002-4a26-a4e8-4525d8f6bee9"
Nov 02 17:29:05.614844 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.614803    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bl7zd\""
Nov 02 17:29:05.614916 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.614869    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 17:29:05.615297 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.615266    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jl8vw\""
Nov 02 17:29:05.615419 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.615406    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 17:29:05.615503 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.615475    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 17:29:05.615612 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.615594    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 17:29:05.615669 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.615637    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 17:29:05.615774 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.615755    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 17:29:05.615859 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.615845    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 17:29:05.616252 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.616234    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 17:29:05.616329 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.616294    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 17:29:05.616329 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.616303    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 17:29:05.616405 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.616364    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qbqb7\""
Nov 02 17:29:05.617168 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.617149    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:05.617286 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:05.617261    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhd7" podUID="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d"
Nov 02 17:29:05.619864 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.619844    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.622650 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.622632    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 17:29:05.622757 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.622667    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:29:05.622757 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.622638    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 17:29:05.622757 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.622745    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 17:29:05.622903 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.622769    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 17:29:05.622903 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.622878    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 17:29:05.622903 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.622891    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 17:29:05.623010 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.622944    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4b9fl\""
Nov 02 17:29:05.624625 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.624362    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 17:29:05.626364 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.626348    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mwsx7\""
Nov 02 17:29:05.627093 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.626926    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xdl26\""
Nov 02 17:29:05.642624 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.642603    2542 reflector.go:446] "Caches populated" logger="kubernetes.io/kubelet-serving" type="*v1.CertificateSigningRequest" reflector="k8s.io/client-go/tools/watch/informerwatcher.go:162"
Nov 02 17:29:05.642713 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.642679    2542 csr.go:274] "Certificate signing request is approved, waiting to be issued" logger="kubernetes.io/kubelet-serving" csr="csr-292sj"
Nov 02 17:29:05.661409 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.661385    2542 csr.go:270] "Certificate signing request is issued" logger="kubernetes.io/kubelet-serving" csr="csr-292sj"
Nov 02 17:29:05.683249 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.683231    2542 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 17:29:05.694803 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.694656    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-sys-fs\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.694803 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.694695    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-netns\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.694803 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.694722    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wsvfn\" (UniqueName: \"kubernetes.io/projected/6767bd29-3165-4e67-81b3-e320e8324034-kube-api-access-wsvfn\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:05.694803 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.694749    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-host\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.694803 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.694774    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-system-cni-dir\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.695050 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.694879    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-run-netns\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.695050 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.694959    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-var-lib-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.695050 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695025    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-cni-bin\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.695167 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695095    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-env-overrides\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.695167 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695124    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-cnibin\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.695244 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695166    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-kubelet\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.695244 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695193    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-daemon-config\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.695244 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695215    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/367d8d85-a5e8-462d-b1d7-91a01e62b726-konnectivity-ca\") pod \"konnectivity-agent-mxsjq\" (UID: \"367d8d85-a5e8-462d-b1d7-91a01e62b726\") " pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:05.695244 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695240    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-cni-multus\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.695378 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695292    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:05.695378 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695320    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:05.695378 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695344    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysctl-d\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.695378 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695360    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-socket-dir-parent\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.695378 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695374    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-k8s-cni-cncf-io\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.695562 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695393    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-systemd\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.695562 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695419    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-systemd\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.695562 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695551    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovn-node-metrics-cert\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.695667 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695586    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5xrlg\" (UniqueName: \"kubernetes.io/projected/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-kube-api-access-5xrlg\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.695667 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695618    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-cni-binary-copy\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.695667 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695652    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnfcz\" (UniqueName: \"kubernetes.io/projected/00b87b36-9ce2-480a-9924-22b33256fe1e-kube-api-access-fnfcz\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:05.695770 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695672    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-log-socket\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.695770 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695693    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/367d8d85-a5e8-462d-b1d7-91a01e62b726-agent-certs\") pod \"konnectivity-agent-mxsjq\" (UID: \"367d8d85-a5e8-462d-b1d7-91a01e62b726\") " pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:05.695770 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695716    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4b72e7f-cc4c-4662-a923-c184069aa118-tmp\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.695770 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695760    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zjwg\" (UniqueName: \"kubernetes.io/projected/ce8b7505-62a5-42f1-b432-d77200c21d2d-kube-api-access-7zjwg\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:05.695940 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695784    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-etc-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.696018 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.695809    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-ovn\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.696075 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696052    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-socket-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.696134 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696091    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-system-cni-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.696134 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696121    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-var-lib-kubelet\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.696285 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696146    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00b87b36-9ce2-480a-9924-22b33256fe1e-host\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:05.696285 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696179    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce8b7505-62a5-42f1-b432-d77200c21d2d-hosts-file\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:05.696285 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696207    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8b7505-62a5-42f1-b432-d77200c21d2d-tmp-dir\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:05.696285 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696234    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.696285 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696258    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-node-log\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.696433 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696330    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tvrzx\" (UniqueName: \"kubernetes.io/projected/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-kube-api-access-tvrzx\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.696433 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696355    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.696433 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696370    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-registration-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.696433 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696397    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6767bd29-3165-4e67-81b3-e320e8324034-iptables-alerter-script\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:05.696561 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696443    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-kubelet\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.696561 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696468    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-slash\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.696561 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696492    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.696561 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696531    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.696683 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696566    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-cni-bin\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.696683 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696580    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-multus-certs\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.696683 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696602    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6767bd29-3165-4e67-81b3-e320e8324034-host-slash\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:05.696683 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696617    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f59q8\" (UniqueName: \"kubernetes.io/projected/f4b72e7f-cc4c-4662-a923-c184069aa118-kube-api-access-f59q8\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.696683 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696631    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cni-binary-copy\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.696683 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696654    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.696683 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696674    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-device-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.696958 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696735    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4g4b6\" (UniqueName: \"kubernetes.io/projected/758467e6-69e4-4c1f-b344-e1e74497c298-kube-api-access-4g4b6\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.696958 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696751    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-conf-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.696958 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696768    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-etc-kubernetes\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.696958 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696785    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-kubelet-config\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:05.696958 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696809    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-modprobe-d\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.696958 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696847    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovnkube-config\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.696958 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696869    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysctl-conf\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.696958 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696894    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-sys\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.696958 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696917    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-lib-modules\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.696958 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696940    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-tuned\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.697360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696962    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-skhvr\" (UniqueName: \"kubernetes.io/projected/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-kube-api-access-skhvr\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.697360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.696985    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00b87b36-9ce2-480a-9924-22b33256fe1e-serviceca\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:05.697360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.697009    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.697360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.697035    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovnkube-script-lib\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.697360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.697059    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-etc-selinux\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.697360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.697084    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-cni-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.697360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.697187    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-hostroot\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.697360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.697213    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-42fn2\" (UniqueName: \"kubernetes.io/projected/dd733778-7002-4a26-a4e8-4525d8f6bee9-kube-api-access-42fn2\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:05.697360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.697236    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-dbus\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:05.697360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.697257    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysconfig\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.697360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.697281    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-systemd-units\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.697360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.697311    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.697360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.697334    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7jnh\" (UniqueName: \"kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh\") pod \"network-check-target-nttxj\" (UID: \"a974c7e3-bf97-4575-bec1-c09fcd018f0d\") " pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:05.697360 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.697350    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-kubernetes\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.698021 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.697370    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-run\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.698021 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.697393    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-cni-netd\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.698021 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.697408    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cnibin\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.715051 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.715016    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 17:29:05.715260 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.715235    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:29:05.715380 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.715296    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 17:29:05.715380 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.715355    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 17:29:05.715488 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.715436    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 17:29:05.715540 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.715519    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vjnnw\""
Nov 02 17:29:05.715590 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.715570    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t2hrp\""
Nov 02 17:29:05.715640 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.715608    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 17:29:05.718941 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.718876    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" event={"ID":"375359043409c490a8ed0d21b8afeaf5","Type":"ContainerStarted","Data":"28fcb0717813a29dc1866541bc3dbb0f0258922adc7924ce1ef6a82d0c101225"}
Nov 02 17:29:05.720251 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.720191    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal" event={"ID":"fbfcd267e5bdda32002f5b72a55d4954","Type":"ContainerStarted","Data":"ee9534b936fc2c51176c28d0ce4ea47c7e9ff43500af65957a17de521caccfe0"}
Nov 02 17:29:05.797651 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.797616    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-socket-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.797853 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.797661    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-system-cni-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.797853 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.797687    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-var-lib-kubelet\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.797853 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.797711    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00b87b36-9ce2-480a-9924-22b33256fe1e-host\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:05.797853 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.797744    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce8b7505-62a5-42f1-b432-d77200c21d2d-hosts-file\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:05.797853 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.797768    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8b7505-62a5-42f1-b432-d77200c21d2d-tmp-dir\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:05.797853 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.797791    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.797853 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.797833    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-node-log\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.798155 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.797861    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvrzx\" (UniqueName: \"kubernetes.io/projected/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-kube-api-access-tvrzx\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.798155 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.797888    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.798155 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.797914    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-registration-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.798155 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.797938    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6767bd29-3165-4e67-81b3-e320e8324034-iptables-alerter-script\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:05.798155 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.797960    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-kubelet\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.798155 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.797982    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-slash\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.798155 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798004    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.798155 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798029    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.798155 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798054    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-cni-bin\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.798155 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798077    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-multus-certs\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.798155 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798099    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6767bd29-3165-4e67-81b3-e320e8324034-host-slash\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:05.798155 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798122    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59q8\" (UniqueName: \"kubernetes.io/projected/f4b72e7f-cc4c-4662-a923-c184069aa118-kube-api-access-f59q8\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.798155 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798145    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cni-binary-copy\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798170    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798192    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-device-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798220    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4b6\" (UniqueName: \"kubernetes.io/projected/758467e6-69e4-4c1f-b344-e1e74497c298-kube-api-access-4g4b6\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798243    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-conf-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798265    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-etc-kubernetes\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798326    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-kubelet-config\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798347    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-modprobe-d\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798362    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovnkube-config\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798376    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysctl-conf\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798389    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-sys\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798411    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-lib-modules\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798428    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-tuned\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798453    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skhvr\" (UniqueName: \"kubernetes.io/projected/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-kube-api-access-skhvr\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798466    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00b87b36-9ce2-480a-9924-22b33256fe1e-serviceca\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798482    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798506    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovnkube-script-lib\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.798726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798522    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-etc-selinux\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798546    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-cni-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798564    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-hostroot\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798587    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42fn2\" (UniqueName: \"kubernetes.io/projected/dd733778-7002-4a26-a4e8-4525d8f6bee9-kube-api-access-42fn2\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798615    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-dbus\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798646    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysconfig\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798670    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-systemd-units\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798688    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798712    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jnh\" (UniqueName: \"kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh\") pod \"network-check-target-nttxj\" (UID: \"a974c7e3-bf97-4575-bec1-c09fcd018f0d\") " pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798738    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-kubernetes\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798762    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-run\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798785    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-cni-netd\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798809    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cnibin\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798849    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-sys-fs\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798875    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-netns\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798902    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsvfn\" (UniqueName: \"kubernetes.io/projected/6767bd29-3165-4e67-81b3-e320e8324034-kube-api-access-wsvfn\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798927    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-host\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.799394 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798956    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-system-cni-dir\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798982    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-run-netns\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.798997    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-var-lib-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799012    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-cni-bin\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799028    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-env-overrides\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799055    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-cnibin\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799080    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-kubelet\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799105    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-daemon-config\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799132    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/367d8d85-a5e8-462d-b1d7-91a01e62b726-konnectivity-ca\") pod \"konnectivity-agent-mxsjq\" (UID: \"367d8d85-a5e8-462d-b1d7-91a01e62b726\") " pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799158    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-cni-multus\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799180    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799208    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799232    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysctl-d\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799250    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-socket-dir-parent\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799269    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-k8s-cni-cncf-io\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799286    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-systemd\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799311    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-systemd\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.800087 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799328    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovn-node-metrics-cert\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799344    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xrlg\" (UniqueName: \"kubernetes.io/projected/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-kube-api-access-5xrlg\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799364    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-cni-binary-copy\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799390    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnfcz\" (UniqueName: \"kubernetes.io/projected/00b87b36-9ce2-480a-9924-22b33256fe1e-kube-api-access-fnfcz\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799411    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-log-socket\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799428    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/367d8d85-a5e8-462d-b1d7-91a01e62b726-agent-certs\") pod \"konnectivity-agent-mxsjq\" (UID: \"367d8d85-a5e8-462d-b1d7-91a01e62b726\") " pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799453    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4b72e7f-cc4c-4662-a923-c184069aa118-tmp\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799481    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zjwg\" (UniqueName: \"kubernetes.io/projected/ce8b7505-62a5-42f1-b432-d77200c21d2d-kube-api-access-7zjwg\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799507    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-etc-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799532    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-ovn\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799624    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-ovn\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799764    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-socket-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799837    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-system-cni-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799888    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-var-lib-kubelet\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799924    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00b87b36-9ce2-480a-9924-22b33256fe1e-host\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.799961    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce8b7505-62a5-42f1-b432-d77200c21d2d-hosts-file\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.800232    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8b7505-62a5-42f1-b432-d77200c21d2d-tmp-dir\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.800387    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.800737 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.800427    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-node-log\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.800924    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-cni-netd\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.800922    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-kubelet\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.800986    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cnibin\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.800992    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-modprobe-d\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801036    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-sys-fs\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801057    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-kubelet-config\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801073    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-slash\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801080    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-netns\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801115    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801144    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801162    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-kubernetes\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801194    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-host\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801201    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801210    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-cni-multus\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801240    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-system-cni-dir\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801265    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-var-lib-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.801545 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801270    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-run-netns\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801307    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-cni-bin\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801360    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysctl-d\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:05.801387    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801400    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-socket-dir-parent\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801430    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-k8s-cni-cncf-io\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:05.801451    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs podName:dd733778-7002-4a26-a4e8-4525d8f6bee9 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:06.30143028 +0000 UTC m=+3.076092533 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs") pod "network-metrics-daemon-tk8ht" (UID: "dd733778-7002-4a26-a4e8-4525d8f6bee9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801458    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-systemd\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801486    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-systemd\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801614    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-device-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801605    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-multus-certs\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801656    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801676    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-conf-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801713    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-cni-bin\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801765    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801788    2542 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801845    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-etc-kubernetes\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801917    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-env-overrides\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.802339 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801922    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6767bd29-3165-4e67-81b3-e320e8324034-host-slash\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801944    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovnkube-config\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801036    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-run\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.801144    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-registration-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:05.802023    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802037    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cni-binary-copy\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802161    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-cnibin\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802304    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysctl-conf\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802342    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-hostroot\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802380    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-sys\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802463    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-lib-modules\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802464    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-log-socket\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802463    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-dbus\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802495    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-kubelet\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802539    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-etc-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:05.802541    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret podName:8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d nodeName:}" failed. No retries permitted until 2026-11-02 17:29:06.302522906 +0000 UTC m=+3.077185162 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret") pod "global-pull-secret-syncer-mmhd7" (UID: "8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802565    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/367d8d85-a5e8-462d-b1d7-91a01e62b726-konnectivity-ca\") pod \"konnectivity-agent-mxsjq\" (UID: \"367d8d85-a5e8-462d-b1d7-91a01e62b726\") " pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:05.803149 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802625    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-etc-selinux\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.803905 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802658    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.803905 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802775    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-cni-binary-copy\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.803905 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802865    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-systemd-units\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.803905 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.802922    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysconfig\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.803905 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.803023    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00b87b36-9ce2-480a-9924-22b33256fe1e-serviceca\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:05.803905 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.803109    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-cni-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.803905 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.803152    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6767bd29-3165-4e67-81b3-e320e8324034-iptables-alerter-script\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:05.803905 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.803159    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovnkube-script-lib\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.803905 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.803211    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-daemon-config\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.806438 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.806416    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-tuned\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.806546 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.806455    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovn-node-metrics-cert\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.806546 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.806536    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/367d8d85-a5e8-462d-b1d7-91a01e62b726-agent-certs\") pod \"konnectivity-agent-mxsjq\" (UID: \"367d8d85-a5e8-462d-b1d7-91a01e62b726\") " pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:05.806767 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.806749    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4b72e7f-cc4c-4662-a923-c184069aa118-tmp\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.828838 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.828769    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skhvr\" (UniqueName: \"kubernetes.io/projected/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-kube-api-access-skhvr\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.835337 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:05.835216    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:29:05.835337 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:05.835241    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:29:05.835337 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:05.835254    2542 projected.go:196] Error preparing data for projected volume kube-api-access-d7jnh for pod openshift-network-diagnostics/network-check-target-nttxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:29:05.835337 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:05.835315    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh podName:a974c7e3-bf97-4575-bec1-c09fcd018f0d nodeName:}" failed. No retries permitted until 2026-11-02 17:29:06.335297807 +0000 UTC m=+3.109960058 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-api-access-d7jnh" (UniqueName: "kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh") pod "network-check-target-nttxj" (UID: "a974c7e3-bf97-4575-bec1-c09fcd018f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:29:05.835604 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.835424    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvrzx\" (UniqueName: \"kubernetes.io/projected/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-kube-api-access-tvrzx\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.838093 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.838068    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnfcz\" (UniqueName: \"kubernetes.io/projected/00b87b36-9ce2-480a-9924-22b33256fe1e-kube-api-access-fnfcz\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:05.838511 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.838488    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59q8\" (UniqueName: \"kubernetes.io/projected/f4b72e7f-cc4c-4662-a923-c184069aa118-kube-api-access-f59q8\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.840727 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.840701    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zjwg\" (UniqueName: \"kubernetes.io/projected/ce8b7505-62a5-42f1-b432-d77200c21d2d-kube-api-access-7zjwg\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:05.842134 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.842111    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42fn2\" (UniqueName: \"kubernetes.io/projected/dd733778-7002-4a26-a4e8-4525d8f6bee9-kube-api-access-42fn2\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:05.842233 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.842176    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsvfn\" (UniqueName: \"kubernetes.io/projected/6767bd29-3165-4e67-81b3-e320e8324034-kube-api-access-wsvfn\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:05.844253 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.844232    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4b6\" (UniqueName: \"kubernetes.io/projected/758467e6-69e4-4c1f-b344-e1e74497c298-kube-api-access-4g4b6\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.844439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.844422    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xrlg\" (UniqueName: \"kubernetes.io/projected/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-kube-api-access-5xrlg\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.908748 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.908720    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:05.916570 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.916381    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:05.923401 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:05.923373    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podce8b7505_62a5_42f1_b432_d77200c21d2d.slice/crio-34704644b2f8aaf65517def9739a5a94a2ef4102e5f63eb6c50786d5d9f25147 WatchSource:0}: Error finding container 34704644b2f8aaf65517def9739a5a94a2ef4102e5f63eb6c50786d5d9f25147: Status 404 returned error can't find the container with id 34704644b2f8aaf65517def9739a5a94a2ef4102e5f63eb6c50786d5d9f25147
Nov 02 17:29:05.929170 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:05.929129    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod00b87b36_9ce2_480a_9924_22b33256fe1e.slice/crio-5539d378b1542324519d761500f1f41db809528c714c434124937ae73d77ecb8 WatchSource:0}: Error finding container 5539d378b1542324519d761500f1f41db809528c714c434124937ae73d77ecb8: Status 404 returned error can't find the container with id 5539d378b1542324519d761500f1f41db809528c714c434124937ae73d77ecb8
Nov 02 17:29:05.941343 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.941311    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:05.942393 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.942359    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:05.947258 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.947238    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:05.953522 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:05.953494    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf26fab1e_e6d4_4caa_bef6_81a09d78c0de.slice/crio-36282a0a35d734aa5ae4edab1c2cfe275e0f3f20e9395e246cb64b674cc64752 WatchSource:0}: Error finding container 36282a0a35d734aa5ae4edab1c2cfe275e0f3f20e9395e246cb64b674cc64752: Status 404 returned error can't find the container with id 36282a0a35d734aa5ae4edab1c2cfe275e0f3f20e9395e246cb64b674cc64752
Nov 02 17:29:05.955100 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:05.955078    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6767bd29_3165_4e67_81b3_e320e8324034.slice/crio-82f83e072dbbfec7616c9c6533f29fc26d7d3ced2d9c309ff133c23d90e397f8 WatchSource:0}: Error finding container 82f83e072dbbfec7616c9c6533f29fc26d7d3ced2d9c309ff133c23d90e397f8: Status 404 returned error can't find the container with id 82f83e072dbbfec7616c9c6533f29fc26d7d3ced2d9c309ff133c23d90e397f8
Nov 02 17:29:05.964418 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.964058    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:05.964802 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:05.964766    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc86c4f40_2978_40f0_bfc1_c3cfffca0d8c.slice/crio-0e45823f1f3e78e26f6c3e2889edb7592f7debd8b8dbcc4dcf99f8f291f90da1 WatchSource:0}: Error finding container 0e45823f1f3e78e26f6c3e2889edb7592f7debd8b8dbcc4dcf99f8f291f90da1: Status 404 returned error can't find the container with id 0e45823f1f3e78e26f6c3e2889edb7592f7debd8b8dbcc4dcf99f8f291f90da1
Nov 02 17:29:05.965773 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.965756    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:05.973764 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.973538    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:05.973832 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:05.973775    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod367d8d85_a5e8_462d_b1d7_91a01e62b726.slice/crio-184ca9e8b8f3f0038d120e2dc91ba930384fad0d99b09baf7cdad8b2b34e0f82 WatchSource:0}: Error finding container 184ca9e8b8f3f0038d120e2dc91ba930384fad0d99b09baf7cdad8b2b34e0f82: Status 404 returned error can't find the container with id 184ca9e8b8f3f0038d120e2dc91ba930384fad0d99b09baf7cdad8b2b34e0f82
Nov 02 17:29:05.975894 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:05.975743    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:05.977037 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:05.977013    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc2d7ba52_9a2b_4871_aca0_3c4e1934c572.slice/crio-39292c76981dbdc83a09b7bd744ad4ab390d9da06ab541901ca5de7d61391dcb WatchSource:0}: Error finding container 39292c76981dbdc83a09b7bd744ad4ab390d9da06ab541901ca5de7d61391dcb: Status 404 returned error can't find the container with id 39292c76981dbdc83a09b7bd744ad4ab390d9da06ab541901ca5de7d61391dcb
Nov 02 17:29:05.983998 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:05.983976    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod758467e6_69e4_4c1f_b344_e1e74497c298.slice/crio-718ae0f8599bafd81a81c85617084005d43c05ef02acf0fca6530b8df0ae5a0f WatchSource:0}: Error finding container 718ae0f8599bafd81a81c85617084005d43c05ef02acf0fca6530b8df0ae5a0f: Status 404 returned error can't find the container with id 718ae0f8599bafd81a81c85617084005d43c05ef02acf0fca6530b8df0ae5a0f
Nov 02 17:29:05.986432 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:05.986409    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4b72e7f_cc4c_4662_a923_c184069aa118.slice/crio-034c0d97f69bd79123d2735864e144d36ae7f0cac80ae88af8d32ad66075330e WatchSource:0}: Error finding container 034c0d97f69bd79123d2735864e144d36ae7f0cac80ae88af8d32ad66075330e: Status 404 returned error can't find the container with id 034c0d97f69bd79123d2735864e144d36ae7f0cac80ae88af8d32ad66075330e
Nov 02 17:29:06.303256 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.303060    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:06.303392 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.303261    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:06.303392 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:06.303225    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:29:06.303392 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:06.303389    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs podName:dd733778-7002-4a26-a4e8-4525d8f6bee9 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:07.303369263 +0000 UTC m=+4.078031513 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs") pod "network-metrics-daemon-tk8ht" (UID: "dd733778-7002-4a26-a4e8-4525d8f6bee9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:29:06.303500 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:06.303396    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:29:06.303500 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:06.303447    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret podName:8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d nodeName:}" failed. No retries permitted until 2026-11-02 17:29:07.303432054 +0000 UTC m=+4.078094307 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret") pod "global-pull-secret-syncer-mmhd7" (UID: "8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:29:06.404140 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.404103    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jnh\" (UniqueName: \"kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh\") pod \"network-check-target-nttxj\" (UID: \"a974c7e3-bf97-4575-bec1-c09fcd018f0d\") " pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:06.404311 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:06.404294    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:29:06.404372 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:06.404315    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:29:06.404372 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:06.404328    2542 projected.go:196] Error preparing data for projected volume kube-api-access-d7jnh for pod openshift-network-diagnostics/network-check-target-nttxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:29:06.404474 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:06.404383    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh podName:a974c7e3-bf97-4575-bec1-c09fcd018f0d nodeName:}" failed. No retries permitted until 2026-11-02 17:29:07.404364978 +0000 UTC m=+4.179027227 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7jnh" (UniqueName: "kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh") pod "network-check-target-nttxj" (UID: "a974c7e3-bf97-4575-bec1-c09fcd018f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:29:06.669018 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.668891    2542 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:24:05 +0000 UTC" deadline="2028-07-01 10:21:34.027531757 +0000 UTC"
Nov 02 17:29:06.669018 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.668971    2542 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="14560h52m27.358565068s"
Nov 02 17:29:06.747848 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.747387    2542 generic.go:356] "Generic (PLEG): container finished" podID="375359043409c490a8ed0d21b8afeaf5" containerID="96cb6a32c416cf30649ef241147265a21068e42498a6ee126aaad556aba5179c" exitCode=0
Nov 02 17:29:06.748194 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.748084    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" event={"ID":"375359043409c490a8ed0d21b8afeaf5","Type":"ContainerDied","Data":"96cb6a32c416cf30649ef241147265a21068e42498a6ee126aaad556aba5179c"}
Nov 02 17:29:06.760527 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.760460    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n9fk9" event={"ID":"f4b72e7f-cc4c-4662-a923-c184069aa118","Type":"ContainerStarted","Data":"034c0d97f69bd79123d2735864e144d36ae7f0cac80ae88af8d32ad66075330e"}
Nov 02 17:29:06.770768 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.770740    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" event={"ID":"758467e6-69e4-4c1f-b344-e1e74497c298","Type":"ContainerStarted","Data":"718ae0f8599bafd81a81c85617084005d43c05ef02acf0fca6530b8df0ae5a0f"}
Nov 02 17:29:06.795513 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.795483    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngkbz" event={"ID":"c2d7ba52-9a2b-4871-aca0-3c4e1934c572","Type":"ContainerStarted","Data":"39292c76981dbdc83a09b7bd744ad4ab390d9da06ab541901ca5de7d61391dcb"}
Nov 02 17:29:06.804342 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.804314    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" event={"ID":"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c","Type":"ContainerStarted","Data":"0e45823f1f3e78e26f6c3e2889edb7592f7debd8b8dbcc4dcf99f8f291f90da1"}
Nov 02 17:29:06.809589 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.809512    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dlp6w" event={"ID":"6767bd29-3165-4e67-81b3-e320e8324034","Type":"ContainerStarted","Data":"82f83e072dbbfec7616c9c6533f29fc26d7d3ced2d9c309ff133c23d90e397f8"}
Nov 02 17:29:06.832071 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.831997    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7qzkz" event={"ID":"f26fab1e-e6d4-4caa-bef6-81a09d78c0de","Type":"ContainerStarted","Data":"36282a0a35d734aa5ae4edab1c2cfe275e0f3f20e9395e246cb64b674cc64752"}
Nov 02 17:29:06.847647 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.847567    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lhwgm" event={"ID":"00b87b36-9ce2-480a-9924-22b33256fe1e","Type":"ContainerStarted","Data":"5539d378b1542324519d761500f1f41db809528c714c434124937ae73d77ecb8"}
Nov 02 17:29:06.868424 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.868293    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal" event={"ID":"fbfcd267e5bdda32002f5b72a55d4954","Type":"ContainerStarted","Data":"032fcf72ec88e49360029b5271b400c196f1b8b14e5eb38e8730bd4f60cb83b2"}
Nov 02 17:29:06.871517 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.871405    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mxsjq" event={"ID":"367d8d85-a5e8-462d-b1d7-91a01e62b726","Type":"ContainerStarted","Data":"184ca9e8b8f3f0038d120e2dc91ba930384fad0d99b09baf7cdad8b2b34e0f82"}
Nov 02 17:29:06.873033 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:06.872916    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hhsrb" event={"ID":"ce8b7505-62a5-42f1-b432-d77200c21d2d","Type":"ContainerStarted","Data":"34704644b2f8aaf65517def9739a5a94a2ef4102e5f63eb6c50786d5d9f25147"}
Nov 02 17:29:07.310892 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:07.310767    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:07.310892 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:07.310841    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:07.311120 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:07.311018    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:29:07.311120 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:07.311092    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret podName:8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d nodeName:}" failed. No retries permitted until 2026-11-02 17:29:09.3110709 +0000 UTC m=+6.085733166 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret") pod "global-pull-secret-syncer-mmhd7" (UID: "8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:29:07.311539 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:07.311517    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:29:07.311600 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:07.311581    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs podName:dd733778-7002-4a26-a4e8-4525d8f6bee9 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:09.311562768 +0000 UTC m=+6.086225011 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs") pod "network-metrics-daemon-tk8ht" (UID: "dd733778-7002-4a26-a4e8-4525d8f6bee9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:29:07.357784 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:07.357624    2542 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:29:07.411341 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:07.411301    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jnh\" (UniqueName: \"kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh\") pod \"network-check-target-nttxj\" (UID: \"a974c7e3-bf97-4575-bec1-c09fcd018f0d\") " pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:07.411558 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:07.411539    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:29:07.411615 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:07.411564    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:29:07.411615 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:07.411578    2542 projected.go:196] Error preparing data for projected volume kube-api-access-d7jnh for pod openshift-network-diagnostics/network-check-target-nttxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:29:07.411713 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:07.411633    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh podName:a974c7e3-bf97-4575-bec1-c09fcd018f0d nodeName:}" failed. No retries permitted until 2026-11-02 17:29:09.411616193 +0000 UTC m=+6.186278444 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7jnh" (UniqueName: "kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh") pod "network-check-target-nttxj" (UID: "a974c7e3-bf97-4575-bec1-c09fcd018f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:29:07.670176 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:07.670035    2542 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:24:05 +0000 UTC" deadline="2028-05-21 00:59:31.29664759 +0000 UTC"
Nov 02 17:29:07.670176 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:07.670098    2542 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="13567h30m23.626555133s"
Nov 02 17:29:07.714670 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:07.713961    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:07.714670 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:07.714105    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8ht" podUID="dd733778-7002-4a26-a4e8-4525d8f6bee9"
Nov 02 17:29:07.714670 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:07.714512    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:07.714670 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:07.714608    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhd7" podUID="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d"
Nov 02 17:29:07.715240 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:07.715098    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:07.715240 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:07.715190    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d"
Nov 02 17:29:07.883045 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:07.882959    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" event={"ID":"375359043409c490a8ed0d21b8afeaf5","Type":"ContainerStarted","Data":"cc6296601e412b777d553d9bc2c1afdc92713fa62371b11edd8c04e5807abdd5"}
Nov 02 17:29:07.925064 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:07.924008    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal" podStartSLOduration=2.9239914110000003 podStartE2EDuration="2.923991411s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:06.911919364 +0000 UTC m=+3.686581623" watchObservedRunningTime="2026-11-02 17:29:07.923991411 +0000 UTC m=+4.698653669"
Nov 02 17:29:08.892371 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:08.892330    2542 generic.go:356] "Generic (PLEG): container finished" podID="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" containerID="1eb8c5ec34c78e0c0caeff100b0da211c968076bca614600a2d6282d3ab5c78c" exitCode=0
Nov 02 17:29:08.893416 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:08.893381    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngkbz" event={"ID":"c2d7ba52-9a2b-4871-aca0-3c4e1934c572","Type":"ContainerDied","Data":"1eb8c5ec34c78e0c0caeff100b0da211c968076bca614600a2d6282d3ab5c78c"}
Nov 02 17:29:08.965018 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:08.964232    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" podStartSLOduration=3.964209922 podStartE2EDuration="3.964209922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:05 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:07.925041863 +0000 UTC m=+4.699704121" watchObservedRunningTime="2026-11-02 17:29:08.964209922 +0000 UTC m=+5.738872207"
Nov 02 17:29:09.326106 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:09.326016    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:09.326106 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:09.326073    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:09.326326 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:09.326189    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:29:09.326326 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:09.326195    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:29:09.326326 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:09.326259    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret podName:8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d nodeName:}" failed. No retries permitted until 2026-11-02 17:29:13.326246313 +0000 UTC m=+10.100908549 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret") pod "global-pull-secret-syncer-mmhd7" (UID: "8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:29:09.326326 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:09.326281    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs podName:dd733778-7002-4a26-a4e8-4525d8f6bee9 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:13.326267912 +0000 UTC m=+10.100930146 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs") pod "network-metrics-daemon-tk8ht" (UID: "dd733778-7002-4a26-a4e8-4525d8f6bee9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:29:09.427077 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:09.427021    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jnh\" (UniqueName: \"kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh\") pod \"network-check-target-nttxj\" (UID: \"a974c7e3-bf97-4575-bec1-c09fcd018f0d\") " pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:09.427264 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:09.427223    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:29:09.427264 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:09.427245    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:29:09.427264 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:09.427257    2542 projected.go:196] Error preparing data for projected volume kube-api-access-d7jnh for pod openshift-network-diagnostics/network-check-target-nttxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:29:09.427420 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:09.427312    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh podName:a974c7e3-bf97-4575-bec1-c09fcd018f0d nodeName:}" failed. No retries permitted until 2026-11-02 17:29:13.427296343 +0000 UTC m=+10.201958585 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7jnh" (UniqueName: "kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh") pod "network-check-target-nttxj" (UID: "a974c7e3-bf97-4575-bec1-c09fcd018f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:29:09.714220 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:09.714180    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:09.714220 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:09.714200    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:09.714468 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:09.714316    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d"
Nov 02 17:29:09.714468 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:09.714380    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:09.714555 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:09.714498    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhd7" podUID="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d"
Nov 02 17:29:09.714648 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:09.714591    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8ht" podUID="dd733778-7002-4a26-a4e8-4525d8f6bee9"
Nov 02 17:29:10.907543 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:10.907455    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/node-ca-lhwgm" event={"ID":"00b87b36-9ce2-480a-9924-22b33256fe1e","Type":"ContainerStarted","Data":"8de289c110641067f5eec70a1ee9554e00a957a3768728ba7ca78bd50b889995"}
Nov 02 17:29:10.910655 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:10.910624    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/konnectivity-agent-mxsjq" event={"ID":"367d8d85-a5e8-462d-b1d7-91a01e62b726","Type":"ContainerStarted","Data":"898855cc4d36932ec91cc8785bbf96aa0e45ef7a7944e53d3a63d48fadcb300e"}
Nov 02 17:29:10.956911 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:10.956854    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/node-ca-lhwgm" podStartSLOduration=3.537523739 podStartE2EDuration="7.956836473s" totalImagesPullingTime="4.419312734s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:10.956261154 +0000 UTC m=+7.730923413" watchObservedRunningTime="2026-11-02 17:29:10.956836473 +0000 UTC m=+7.731498727"
Nov 02 17:29:11.716477 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:11.716447    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:11.716625 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:11.716579    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d"
Nov 02 17:29:11.716625 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:11.716613    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:11.716750 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:11.716730    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8ht" podUID="dd733778-7002-4a26-a4e8-4525d8f6bee9"
Nov 02 17:29:11.716802 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:11.716793    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:11.717185 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:11.716873    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhd7" podUID="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d"
Nov 02 17:29:11.915868 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:11.915836    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" event={"ID":"758467e6-69e4-4c1f-b344-e1e74497c298","Type":"ContainerStarted","Data":"ad864ff6740e4c778f72c8079e4f03eacc7840d208ff11512bdc1dc1655e8c7a"}
Nov 02 17:29:12.928320 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:12.928290    2542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:13.281878 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.281836    2542 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 17:29:13.364439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.363943    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:13.364439 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.363984    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:13.364439 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:13.364068    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:29:13.364439 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:13.364108    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret podName:8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d nodeName:}" failed. No retries permitted until 2026-11-02 17:29:21.364097114 +0000 UTC m=+18.138759349 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret") pod "global-pull-secret-syncer-mmhd7" (UID: "8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:29:13.364439 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:13.364384    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:29:13.364439 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:13.364416    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs podName:dd733778-7002-4a26-a4e8-4525d8f6bee9 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:21.36440735 +0000 UTC m=+18.139069584 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs") pod "network-metrics-daemon-tk8ht" (UID: "dd733778-7002-4a26-a4e8-4525d8f6bee9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:29:13.465282 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.464636    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jnh\" (UniqueName: \"kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh\") pod \"network-check-target-nttxj\" (UID: \"a974c7e3-bf97-4575-bec1-c09fcd018f0d\") " pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:13.465282 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:13.464845    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:29:13.465282 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:13.464870    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:29:13.465282 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:13.464883    2542 projected.go:196] Error preparing data for projected volume kube-api-access-d7jnh for pod openshift-network-diagnostics/network-check-target-nttxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:29:13.465282 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:13.464950    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh podName:a974c7e3-bf97-4575-bec1-c09fcd018f0d nodeName:}" failed. No retries permitted until 2026-11-02 17:29:21.464931154 +0000 UTC m=+18.239593412 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7jnh" (UniqueName: "kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh") pod "network-check-target-nttxj" (UID: "a974c7e3-bf97-4575-bec1-c09fcd018f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:29:13.649426 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.648504    2542 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:29:13.281857926Z","UUID":"187fe16a-869d-436c-890c-bc029a710a17","Handler":null,"Name":"","Endpoint":""}
Nov 02 17:29:13.650369 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.650240    2542 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 17:29:13.650369 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.650277    2542 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 17:29:13.714076 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.713936    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:13.714076 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:13.714056    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d"
Nov 02 17:29:13.715118 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.715096    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:13.715224 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:13.715198    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8ht" podUID="dd733778-7002-4a26-a4e8-4525d8f6bee9"
Nov 02 17:29:13.715397 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.715346    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:13.715473 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:13.715433    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhd7" podUID="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d"
Nov 02 17:29:13.923128 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.922900    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" event={"ID":"758467e6-69e4-4c1f-b344-e1e74497c298","Type":"ContainerStarted","Data":"2f2d0ecef34f90d7be65ae3bfde69eebfd2e056a726c73ec5c4cc23ac48cb253"}
Nov 02 17:29:13.926762 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.926738    2542 generic.go:356] "Generic (PLEG): container finished" podID="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" containerID="5ca757f8bc481eca00d6d4427507cd91996755dcb73a5bf277184dde2a3f42ef" exitCode=0
Nov 02 17:29:13.926889 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.926842    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngkbz" event={"ID":"c2d7ba52-9a2b-4871-aca0-3c4e1934c572","Type":"ContainerDied","Data":"5ca757f8bc481eca00d6d4427507cd91996755dcb73a5bf277184dde2a3f42ef"}
Nov 02 17:29:13.932176 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.932147    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-7qzkz" event={"ID":"f26fab1e-e6d4-4caa-bef6-81a09d78c0de","Type":"ContainerStarted","Data":"7803aa0124b2e9537d735124cfa4c0868e0e9caf60d8681a45cb7d45ad3926e7"}
Nov 02 17:29:13.934680 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.934657    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/node-resolver-hhsrb" event={"ID":"ce8b7505-62a5-42f1-b432-d77200c21d2d","Type":"ContainerStarted","Data":"76eb2397438267073fbae51cb83275becfa2eb6b5e76ece4561a9e16d797fe7f"}
Nov 02 17:29:13.994235 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:13.993639    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kube-system/konnectivity-agent-mxsjq" podStartSLOduration=7.232346944 podStartE2EDuration="10.993620543s" totalImagesPullingTime="3.761273599s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:10.998435871 +0000 UTC m=+7.773098129" watchObservedRunningTime="2026-11-02 17:29:13.993620543 +0000 UTC m=+10.768282801"
Nov 02 17:29:14.168923 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:14.168872    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-dns/node-resolver-hhsrb" podStartSLOduration=3.371659462 podStartE2EDuration="11.168856314s" totalImagesPullingTime="7.797196852s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:14.120272487 +0000 UTC m=+10.894934757" watchObservedRunningTime="2026-11-02 17:29:14.168856314 +0000 UTC m=+10.943518626"
Nov 02 17:29:14.939986 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:14.939162    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" event={"ID":"758467e6-69e4-4c1f-b344-e1e74497c298","Type":"ContainerStarted","Data":"82f299a64b1e139bf279062152dead44b2931bf0cae71f836854e3afc76f09fb"}
Nov 02 17:29:14.942859 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:14.942830    2542 generic.go:356] "Generic (PLEG): container finished" podID="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" containerID="f375f7cea091ed446006811bbcf4e8902d0c6b9bf4b3e5f6d6ba9fd8cc34e740" exitCode=0
Nov 02 17:29:14.942981 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:14.942914    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngkbz" event={"ID":"c2d7ba52-9a2b-4871-aca0-3c4e1934c572","Type":"ContainerDied","Data":"f375f7cea091ed446006811bbcf4e8902d0c6b9bf4b3e5f6d6ba9fd8cc34e740"}
Nov 02 17:29:14.983448 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:14.983396    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-7qzkz" podStartSLOduration=4.698330331 podStartE2EDuration="11.983379485s" totalImagesPullingTime="7.285049154s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:14.169265583 +0000 UTC m=+10.943927840" watchObservedRunningTime="2026-11-02 17:29:14.983379485 +0000 UTC m=+11.758041745"
Nov 02 17:29:14.983608 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:14.983555    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podStartSLOduration=3.206670977 podStartE2EDuration="10.983548638s" totalImagesPullingTime="7.776877661s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:04 +0000 UTC" imagePullSessionsCount=3 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:14.982945633 +0000 UTC m=+11.757607889" watchObservedRunningTime="2026-11-02 17:29:14.983548638 +0000 UTC m=+11.758210925"
Nov 02 17:29:15.307977 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:15.307947    2542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:15.308893 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:15.308692    2542 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:15.714717 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:15.714682    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:15.714910 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:15.714691    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:15.714910 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:15.714828    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:15.714910 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:15.714833    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d"
Nov 02 17:29:15.715076 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:15.714916    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhd7" podUID="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d"
Nov 02 17:29:15.715076 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:15.715017    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8ht" podUID="dd733778-7002-4a26-a4e8-4525d8f6bee9"
Nov 02 17:29:15.952648 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:15.952613    2542 generic.go:356] "Generic (PLEG): container finished" podID="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" containerID="3c53a4dfddf899285adb42dc751a3c0735ffe894c1506754ee9c0e85238f9d1f" exitCode=0
Nov 02 17:29:15.953129 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:15.952713    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngkbz" event={"ID":"c2d7ba52-9a2b-4871-aca0-3c4e1934c572","Type":"ContainerDied","Data":"3c53a4dfddf899285adb42dc751a3c0735ffe894c1506754ee9c0e85238f9d1f"}
Nov 02 17:29:15.954886 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:15.954843    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-operator/iptables-alerter-dlp6w" event={"ID":"6767bd29-3165-4e67-81b3-e320e8324034","Type":"ContainerStarted","Data":"845ea0681b317d3c1ef6dc05cd4e200ffd68f9225adda17ddb5397275608c503"}
Nov 02 17:29:15.955993 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:15.955972    2542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:16.115398 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:16.115343    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-operator/iptables-alerter-dlp6w" podStartSLOduration=5.189829931 podStartE2EDuration="13.115328283s" totalImagesPullingTime="7.925498352s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:16.046339925 +0000 UTC m=+12.821002182" watchObservedRunningTime="2026-11-02 17:29:16.115328283 +0000 UTC m=+12.889990540"
Nov 02 17:29:16.959118 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:16.959069    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-node-tuning-operator/tuned-n9fk9" event={"ID":"f4b72e7f-cc4c-4662-a923-c184069aa118","Type":"ContainerStarted","Data":"cf5f894c0077d88ff306f393cc30e6874faf0bdfe8116ed435b74d7f517bba76"}
Nov 02 17:29:17.003530 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:17.003473    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-cluster-node-tuning-operator/tuned-n9fk9" podStartSLOduration=3.1024815 podStartE2EDuration="13.003456273s" totalImagesPullingTime="9.900974773s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:04 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:17.002870343 +0000 UTC m=+13.777532601" watchObservedRunningTime="2026-11-02 17:29:17.003456273 +0000 UTC m=+13.778118531"
Nov 02 17:29:17.714587 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:17.714557    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:17.714715 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:17.714562    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:17.714715 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:17.714670    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d"
Nov 02 17:29:17.714715 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:17.714698    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:17.714898 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:17.714797    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8ht" podUID="dd733778-7002-4a26-a4e8-4525d8f6bee9"
Nov 02 17:29:17.715013 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:17.714984    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhd7" podUID="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d"
Nov 02 17:29:17.967341 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:17.967013    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngkbz" event={"ID":"c2d7ba52-9a2b-4871-aca0-3c4e1934c572","Type":"ContainerDied","Data":"859ebbb72520ce950d67654afed6c62c3bfe8c7861db1d0485a4ca0a53d9a58c"}
Nov 02 17:29:17.967845 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:17.967054    2542 generic.go:356] "Generic (PLEG): container finished" podID="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" containerID="859ebbb72520ce950d67654afed6c62c3bfe8c7861db1d0485a4ca0a53d9a58c" exitCode=0
Nov 02 17:29:18.973199 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:18.973158    2542 generic.go:356] "Generic (PLEG): container finished" podID="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" containerID="d5407571491be8eb6db5a9af854fe88a1ba9d3cab526078d243a673491607c1f" exitCode=0
Nov 02 17:29:18.973649 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:18.973222    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngkbz" event={"ID":"c2d7ba52-9a2b-4871-aca0-3c4e1934c572","Type":"ContainerDied","Data":"d5407571491be8eb6db5a9af854fe88a1ba9d3cab526078d243a673491607c1f"}
Nov 02 17:29:19.714017 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:19.713982    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:19.714017 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:19.713996    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:19.714229 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:19.714121    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d"
Nov 02 17:29:19.714229 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:19.714187    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8ht" podUID="dd733778-7002-4a26-a4e8-4525d8f6bee9"
Nov 02 17:29:19.714350 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:19.714227    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:19.714350 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:19.714330    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhd7" podUID="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d"
Nov 02 17:29:19.977868 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:19.977793    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/multus-additional-cni-plugins-ngkbz" event={"ID":"c2d7ba52-9a2b-4871-aca0-3c4e1934c572","Type":"ContainerStarted","Data":"77bb9f2ad50829f9a318931ff8ce14ddd9cb1d6f9bc7a1552ed94b97a4cdcafe"}
Nov 02 17:29:20.080259 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:20.080209    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-multus/multus-additional-cni-plugins-ngkbz" podStartSLOduration=7.205260941 podStartE2EDuration="17.080195017s" totalImagesPullingTime="9.874934076s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:03 +0000 UTC" imagePullSessionsCount=5 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:20.080097431 +0000 UTC m=+16.854759703" watchObservedRunningTime="2026-11-02 17:29:20.080195017 +0000 UTC m=+16.854857275"
Nov 02 17:29:20.982702 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:20.982535    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" event={"ID":"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c","Type":"ContainerStarted","Data":"1007a8808e75a6c28281368531bd3f6683b101e3823373cdf7a5109c915acfca"}
Nov 02 17:29:20.983079 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:20.982765    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" event={"ID":"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c","Type":"ContainerStarted","Data":"0e0c7ab79ffb2c085b7e9859cae61d855dbd9c991fbec865e4023befb01e6e1a"}
Nov 02 17:29:20.983079 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:20.982797    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" event={"ID":"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c","Type":"ContainerStarted","Data":"f707b441d8a05d9087bd50b5d15efd63edce36ffeaf03a1c26eb5c48294ccfc4"}
Nov 02 17:29:20.983079 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:20.982838    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" event={"ID":"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c","Type":"ContainerStarted","Data":"296ea4ef8869d34f06182e320b569ce1ea5fe28eea446d3679265720d169b446"}
Nov 02 17:29:20.983079 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:20.982870    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" event={"ID":"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c","Type":"ContainerStarted","Data":"1f57e5720c698b40649a607fd8434af1af9e0e5f426245d2cf8cd312cd82e5da"}
Nov 02 17:29:20.983079 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:20.982902    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" event={"ID":"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c","Type":"ContainerStarted","Data":"b24db19925c4a3a3a499c083c2c66d95bec69e9c98509d2a20fd38076666b6cc"}
Nov 02 17:29:21.427066 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:21.427039    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:21.427066 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:21.427068    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:21.427294 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:21.427186    2542 secret.go:189] Couldn't get secret kube-system/original-pull-secret: object "kube-system"/"original-pull-secret" not registered
Nov 02 17:29:21.427294 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:21.427186    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:29:21.427294 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:21.427251    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret podName:8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d nodeName:}" failed. No retries permitted until 2026-11-02 17:29:37.427234857 +0000 UTC m=+34.201897092 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "original-pull-secret" (UniqueName: "kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret") pod "global-pull-secret-syncer-mmhd7" (UID: "8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d") : object "kube-system"/"original-pull-secret" not registered
Nov 02 17:29:21.427294 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:21.427267    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs podName:dd733778-7002-4a26-a4e8-4525d8f6bee9 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:37.42726024 +0000 UTC m=+34.201922474 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs") pod "network-metrics-daemon-tk8ht" (UID: "dd733778-7002-4a26-a4e8-4525d8f6bee9") : object "openshift-multus"/"metrics-daemon-secret" not registered
Nov 02 17:29:21.528135 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:21.528105    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jnh\" (UniqueName: \"kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh\") pod \"network-check-target-nttxj\" (UID: \"a974c7e3-bf97-4575-bec1-c09fcd018f0d\") " pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:21.528265 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:21.528248    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/kube-root-ca.crt: object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered
Nov 02 17:29:21.528303 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:21.528270    2542 projected.go:291] Couldn't get configMap openshift-network-diagnostics/openshift-service-ca.crt: object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered
Nov 02 17:29:21.528303 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:21.528278    2542 projected.go:196] Error preparing data for projected volume kube-api-access-d7jnh for pod openshift-network-diagnostics/network-check-target-nttxj: [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:29:21.528357 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:21.528319    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh podName:a974c7e3-bf97-4575-bec1-c09fcd018f0d nodeName:}" failed. No retries permitted until 2026-11-02 17:29:37.528307833 +0000 UTC m=+34.302970104 (durationBeforeRetry 16s). Error: MountVolume.SetUp failed for volume "kube-api-access-d7jnh" (UniqueName: "kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh") pod "network-check-target-nttxj" (UID: "a974c7e3-bf97-4575-bec1-c09fcd018f0d") : [object "openshift-network-diagnostics"/"kube-root-ca.crt" not registered, object "openshift-network-diagnostics"/"openshift-service-ca.crt" not registered]
Nov 02 17:29:21.713673 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:21.713650    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:21.713855 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:21.713650    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:21.713855 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:21.713746    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhd7" podUID="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d"
Nov 02 17:29:21.713855 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:21.713766    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:21.713999 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:21.713870    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8ht" podUID="dd733778-7002-4a26-a4e8-4525d8f6bee9"
Nov 02 17:29:21.713999 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:21.713981    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d"
Nov 02 17:29:22.990171 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:22.990132    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" event={"ID":"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c","Type":"ContainerStarted","Data":"c85e2174c667b93c00a4c61bce54d357ace38a16f087c31c6415b9ccbc761f92"}
Nov 02 17:29:23.715375 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:23.715340    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:23.715528 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:23.715423    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:23.715528 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:23.715469    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:23.715528 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:23.715464    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d"
Nov 02 17:29:23.715641 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:23.715553    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhd7" podUID="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d"
Nov 02 17:29:23.715671 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:23.715643    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8ht" podUID="dd733778-7002-4a26-a4e8-4525d8f6bee9"
Nov 02 17:29:24.998666 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:24.998626    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" event={"ID":"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c","Type":"ContainerStarted","Data":"d8f0d9bcc31bfb03128cf8096922bcabc511ffd73d885355f5b2ae417aa2bb93"}
Nov 02 17:29:24.999336 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:24.999082    2542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:24.999336 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:24.999152    2542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:25.016282 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:25.016260    2542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:25.020397 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:25.020375    2542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:25.089892 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:25.089839    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" podStartSLOduration=7.5697660429999996 podStartE2EDuration="22.089801466s" totalImagesPullingTime="14.520035423s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 17:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:25.089298789 +0000 UTC m=+21.863961046" watchObservedRunningTime="2026-11-02 17:29:25.089801466 +0000 UTC m=+21.864463735"
Nov 02 17:29:25.713752 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:25.713713    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:25.713968 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:25.713725    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:25.713968 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:25.713851    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:25.713968 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:25.713848    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d"
Nov 02 17:29:25.713968 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:25.713929    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8ht" podUID="dd733778-7002-4a26-a4e8-4525d8f6bee9"
Nov 02 17:29:25.714165 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:25.713993    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhd7" podUID="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d"
Nov 02 17:29:26.000454 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:26.000394    2542 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 17:29:27.002298 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:27.002232    2542 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 17:29:27.056749 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:27.056722    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mmhd7"]
Nov 02 17:29:27.056906 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:27.056839    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:27.056945 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:27.056927    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhd7" podUID="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d"
Nov 02 17:29:27.061617 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:27.061595    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nttxj"]
Nov 02 17:29:27.061700 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:27.061671    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:27.061759 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:27.061744    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d"
Nov 02 17:29:27.068613 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:27.068593    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tk8ht"]
Nov 02 17:29:27.068682 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:27.068666    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:27.068748 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:27.068733    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8ht" podUID="dd733778-7002-4a26-a4e8-4525d8f6bee9"
Nov 02 17:29:28.681954 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:28.681919    2542 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:28.714379 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:28.714350    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:28.714379 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:28.714365    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:28.714379 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:28.714375    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:28.714611 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:28.714460    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhd7" podUID="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d"
Nov 02 17:29:28.714611 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:28.714508    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d"
Nov 02 17:29:28.714611 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:28.714585    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8ht" podUID="dd733778-7002-4a26-a4e8-4525d8f6bee9"
Nov 02 17:29:30.713961 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:30.713928    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:30.713961 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:30.713943    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:30.714557 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:30.714060    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhd7" podUID="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d"
Nov 02 17:29:30.714557 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:30.714105    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8ht" podUID="dd733778-7002-4a26-a4e8-4525d8f6bee9"
Nov 02 17:29:30.714557 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:30.714142    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:30.714557 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:30.714206    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d"
Nov 02 17:29:31.008595 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:31.008543    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:31.318027 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:31.317957    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:32.714247 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:32.714211    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:32.714668 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:32.714249    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:32.714668 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:32.714311    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-multus/network-metrics-daemon-tk8ht" podUID="dd733778-7002-4a26-a4e8-4525d8f6bee9"
Nov 02 17:29:32.714668 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:32.714342    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:32.714668 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:32.714427    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d"
Nov 02 17:29:32.714668 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:32.714494    2542 pod_workers.go:1324] "Error syncing pod, skipping" err="network is not ready: container runtime network not ready: NetworkReady=false reason:NetworkPluginNotReady message:Network plugin returns error: no CNI configuration file in /etc/kubernetes/cni/net.d/. Has your network provider started?" pod="kube-system/global-pull-secret-syncer-mmhd7" podUID="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d"
Nov 02 17:29:33.068852 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.068827    2542 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeReady"
Nov 02 17:29:33.068997 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.068936    2542 kubelet_node_status.go:477] "Fast updating node status as it just became ready"
Nov 02 17:29:33.138950 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.138926    2542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress-canary/ingress-canary-n4r8t"]
Nov 02 17:29:33.146196 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.146164    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:33.184805 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.184782    2542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-dns/dns-default-zmrdt"]
Nov 02 17:29:33.184954 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.184907    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n4r8t"
Nov 02 17:29:33.194376 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.194356    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n4r8t"]
Nov 02 17:29:33.194458 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.194389    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zmrdt"]
Nov 02 17:29:33.194490 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.194460    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:33.202330 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.202309    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 17:29:33.202704 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.202686    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rj9zp\""
Nov 02 17:29:33.202704 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.202702    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 17:29:33.202885 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.202705    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 17:29:33.208909 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.208893    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 17:29:33.214770 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.214754    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 17:29:33.214878 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.214791    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5nstj\""
Nov 02 17:29:33.314155 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.313910    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kfzh6\" (UniqueName: \"kubernetes.io/projected/ba5cbec9-7649-462a-b110-11ebcc5d6d78-kube-api-access-kfzh6\") pod \"ingress-canary-n4r8t\" (UID: \"ba5cbec9-7649-462a-b110-11ebcc5d6d78\") " pod="openshift-ingress-canary/ingress-canary-n4r8t"
Nov 02 17:29:33.314412 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.314384    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f59eba5d-a500-462e-9ecd-4ef9e553f873-config-volume\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:33.314476 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.314463    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba5cbec9-7649-462a-b110-11ebcc5d6d78-cert\") pod \"ingress-canary-n4r8t\" (UID: \"ba5cbec9-7649-462a-b110-11ebcc5d6d78\") " pod="openshift-ingress-canary/ingress-canary-n4r8t"
Nov 02 17:29:33.314543 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.314529    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:33.314591 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.314561    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59eba5d-a500-462e-9ecd-4ef9e553f873-tmp-dir\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:33.314635 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.314593    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbbbk\" (UniqueName: \"kubernetes.io/projected/f59eba5d-a500-462e-9ecd-4ef9e553f873-kube-api-access-cbbbk\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:33.415304 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.415272    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfzh6\" (UniqueName: \"kubernetes.io/projected/ba5cbec9-7649-462a-b110-11ebcc5d6d78-kube-api-access-kfzh6\") pod \"ingress-canary-n4r8t\" (UID: \"ba5cbec9-7649-462a-b110-11ebcc5d6d78\") " pod="openshift-ingress-canary/ingress-canary-n4r8t"
Nov 02 17:29:33.415304 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.415306    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f59eba5d-a500-462e-9ecd-4ef9e553f873-config-volume\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:33.415519 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.415343    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba5cbec9-7649-462a-b110-11ebcc5d6d78-cert\") pod \"ingress-canary-n4r8t\" (UID: \"ba5cbec9-7649-462a-b110-11ebcc5d6d78\") " pod="openshift-ingress-canary/ingress-canary-n4r8t"
Nov 02 17:29:33.415577 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.415518    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:33.415577 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.415556    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59eba5d-a500-462e-9ecd-4ef9e553f873-tmp-dir\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:33.415675 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.415583    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbbbk\" (UniqueName: \"kubernetes.io/projected/f59eba5d-a500-462e-9ecd-4ef9e553f873-kube-api-access-cbbbk\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:33.415725 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:33.415676    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:29:33.415773 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:33.415750    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls podName:f59eba5d-a500-462e-9ecd-4ef9e553f873 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:33.915731028 +0000 UTC m=+30.690393276 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls") pod "dns-default-zmrdt" (UID: "f59eba5d-a500-462e-9ecd-4ef9e553f873") : secret "dns-default-metrics-tls" not found
Nov 02 17:29:33.415934 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.415909    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59eba5d-a500-462e-9ecd-4ef9e553f873-tmp-dir\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:33.416011 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.415984    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f59eba5d-a500-462e-9ecd-4ef9e553f873-config-volume\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:33.419670 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.419649    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba5cbec9-7649-462a-b110-11ebcc5d6d78-cert\") pod \"ingress-canary-n4r8t\" (UID: \"ba5cbec9-7649-462a-b110-11ebcc5d6d78\") " pod="openshift-ingress-canary/ingress-canary-n4r8t"
Nov 02 17:29:33.440147 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.440126    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbbbk\" (UniqueName: \"kubernetes.io/projected/f59eba5d-a500-462e-9ecd-4ef9e553f873-kube-api-access-cbbbk\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:33.440925 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.440902    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfzh6\" (UniqueName: \"kubernetes.io/projected/ba5cbec9-7649-462a-b110-11ebcc5d6d78-kube-api-access-kfzh6\") pod \"ingress-canary-n4r8t\" (UID: \"ba5cbec9-7649-462a-b110-11ebcc5d6d78\") " pod="openshift-ingress-canary/ingress-canary-n4r8t"
Nov 02 17:29:33.502800 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.502772    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress-canary/ingress-canary-n4r8t"
Nov 02 17:29:33.509570 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.509545    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:33.727544 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.727492    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:33.866742 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.866658    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress-canary/ingress-canary-n4r8t"]
Nov 02 17:29:33.870723 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:33.870697    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podba5cbec9_7649_462a_b110_11ebcc5d6d78.slice/crio-463c6701469021f7bcc9b98670fa5c431575005397ec36ed2c36164ecc754824 WatchSource:0}: Error finding container 463c6701469021f7bcc9b98670fa5c431575005397ec36ed2c36164ecc754824: Status 404 returned error can't find the container with id 463c6701469021f7bcc9b98670fa5c431575005397ec36ed2c36164ecc754824
Nov 02 17:29:33.920678 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:33.920649    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:33.920831 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:33.920752    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:29:33.920831 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:33.920797    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls podName:f59eba5d-a500-462e-9ecd-4ef9e553f873 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:34.920784885 +0000 UTC m=+31.695447121 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls") pod "dns-default-zmrdt" (UID: "f59eba5d-a500-462e-9ecd-4ef9e553f873") : secret "dns-default-metrics-tls" not found
Nov 02 17:29:34.019242 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:34.019206    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n4r8t" event={"ID":"ba5cbec9-7649-462a-b110-11ebcc5d6d78","Type":"ContainerStarted","Data":"463c6701469021f7bcc9b98670fa5c431575005397ec36ed2c36164ecc754824"}
Nov 02 17:29:34.714426 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:34.714390    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:34.714828 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:34.714798    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:34.715214 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:34.715188    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:34.730529 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:34.730462    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 17:29:34.730529 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:34.730486    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 17:29:34.730995 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:34.730737    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zt2z5\""
Nov 02 17:29:34.732568 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:34.732551    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 17:29:34.732681 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:34.732665    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 17:29:34.732863 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:34.732847    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ppwdb\""
Nov 02 17:29:34.743195 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:34.743171    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn2p2\""
Nov 02 17:29:34.928965 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:34.928929    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:34.929145 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:34.929089    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:29:34.929193 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:34.929150    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls podName:f59eba5d-a500-462e-9ecd-4ef9e553f873 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:36.929131159 +0000 UTC m=+33.703793397 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls") pod "dns-default-zmrdt" (UID: "f59eba5d-a500-462e-9ecd-4ef9e553f873") : secret "dns-default-metrics-tls" not found
Nov 02 17:29:35.106579 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:35.106506    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:35.247145 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:35.247115    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:36.944072 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:36.944041    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:36.944429 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:36.944175    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:29:36.944429 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:36.944233    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls podName:f59eba5d-a500-462e-9ecd-4ef9e553f873 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:40.944220624 +0000 UTC m=+37.718882859 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls") pod "dns-default-zmrdt" (UID: "f59eba5d-a500-462e-9ecd-4ef9e553f873") : secret "dns-default-metrics-tls" not found
Nov 02 17:29:37.026920 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.026882    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress-canary/ingress-canary-n4r8t" event={"ID":"ba5cbec9-7649-462a-b110-11ebcc5d6d78","Type":"ContainerStarted","Data":"32540665d398fabb025e03bf23cc30d995e21eae2bc314faaa6848ac39cdeddc"}
Nov 02 17:29:37.068847 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.068777    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress-canary/ingress-canary-n4r8t" podStartSLOduration=1.336718433 podStartE2EDuration="4.068762123s" totalImagesPullingTime="2.73204369s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:33 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:37.068731818 +0000 UTC m=+33.843394074" watchObservedRunningTime="2026-11-02 17:29:37.068762123 +0000 UTC m=+33.843424370"
Nov 02 17:29:37.447730 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.447697    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:37.447941 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.447736    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:37.447941 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:37.447872    2542 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:29:37.448064 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:37.447949    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs podName:dd733778-7002-4a26-a4e8-4525d8f6bee9 nodeName:}" failed. No retries permitted until 2026-11-02 17:30:09.447927671 +0000 UTC m=+66.222589922 (durationBeforeRetry 32s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs") pod "network-metrics-daemon-tk8ht" (UID: "dd733778-7002-4a26-a4e8-4525d8f6bee9") : secret "metrics-daemon-secret" not found
Nov 02 17:29:37.450103 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.450076    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:37.459068 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.459045    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:37.548782 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.548754    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jnh\" (UniqueName: \"kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh\") pod \"network-check-target-nttxj\" (UID: \"a974c7e3-bf97-4575-bec1-c09fcd018f0d\") " pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:37.551558 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.551527    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7jnh\" (UniqueName: \"kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh\") pod \"network-check-target-nttxj\" (UID: \"a974c7e3-bf97-4575-bec1-c09fcd018f0d\") " pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:37.597129 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.597103    2542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/insights-runtime-extractor-6bfcf"]
Nov 02 17:29:37.624084 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.624061    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:37.627830 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.627782    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6bfcf"]
Nov 02 17:29:37.659632 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.659601    2542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-insights/periodic-gathering-wphg6-wf57n"]
Nov 02 17:29:37.673805 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.673784    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:37.682867 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.682843    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wphg6-wf57n"]
Nov 02 17:29:37.695828 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.695796    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:37.720565 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.720525    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 17:29:37.721083 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.721061    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-c5qgh\""
Nov 02 17:29:37.721083 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.721061    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fw7vt\""
Nov 02 17:29:37.722127 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.722110    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 17:29:37.722127 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.722123    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 17:29:37.722241 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.722148    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 17:29:37.722241 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.722150    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 17:29:37.746646 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.746629    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 17:29:37.750577 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.750550    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4kfc\" (UniqueName: \"kubernetes.io/projected/83455bdf-65dc-4f66-b943-55835e92b360-kube-api-access-l4kfc\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:37.750668 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.750592    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/83455bdf-65dc-4f66-b943-55835e92b360-archives-path\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:37.750668 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.750619    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-trusted-ca-bundle\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:37.750759 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.750671    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-data-volume\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:37.750808 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.750788    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:37.750889 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.750877    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:37.750937 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.750915    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-crio-socket\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:37.750973 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.750965    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-service-ca-bundle\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:37.751005 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.750984    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kd2rw\" (UniqueName: \"kubernetes.io/projected/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-api-access-kd2rw\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:37.778749 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.778731    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:37.831533 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.831500    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kube-system/global-pull-secret-syncer-mmhd7"]
Nov 02 17:29:37.835047 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:37.835013    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8fe19c0f_fd25_49b6_bb3a_9ad60c173c9d.slice/crio-f4fb571e9cb3fe2a0f336911a73b85beae343bd4305c83c039786b15981b271e WatchSource:0}: Error finding container f4fb571e9cb3fe2a0f336911a73b85beae343bd4305c83c039786b15981b271e: Status 404 returned error can't find the container with id f4fb571e9cb3fe2a0f336911a73b85beae343bd4305c83c039786b15981b271e
Nov 02 17:29:37.852546 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.852462    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:37.852546 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.852538    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:37.852700 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.852565    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-crio-socket\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:37.852700 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.852608    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-service-ca-bundle\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:37.852700 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.852632    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd2rw\" (UniqueName: \"kubernetes.io/projected/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-api-access-kd2rw\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:37.852700 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.852678    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4kfc\" (UniqueName: \"kubernetes.io/projected/83455bdf-65dc-4f66-b943-55835e92b360-kube-api-access-l4kfc\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:37.852907 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.852712    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/83455bdf-65dc-4f66-b943-55835e92b360-archives-path\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:37.852907 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.852729    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-trusted-ca-bundle\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:37.852907 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.852744    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-data-volume\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:37.853733 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:37.853412    2542 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:29:37.853733 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:37.853479    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm podName:da9f1b66-0e1b-4c7e-a947-a897c00ccc19 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:38.353463031 +0000 UTC m=+35.128125284 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6bfcf" (UID: "da9f1b66-0e1b-4c7e-a947-a897c00ccc19") : configmap "kube-rbac-proxy" not found
Nov 02 17:29:37.853733 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.853477    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-data-volume\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:37.853733 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:37.853565    2542 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:37.853733 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:37.853602    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls podName:da9f1b66-0e1b-4c7e-a947-a897c00ccc19 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:38.353589985 +0000 UTC m=+35.128252237 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6bfcf" (UID: "da9f1b66-0e1b-4c7e-a947-a897c00ccc19") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:37.853733 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.853686    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-crio-socket\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:37.853733 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.853722    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/83455bdf-65dc-4f66-b943-55835e92b360-archives-path\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:37.854094 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.854017    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-service-ca-bundle\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:37.855009 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.854990    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-trusted-ca-bundle\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:37.855434 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.855416    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:37.879363 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.879342    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4kfc\" (UniqueName: \"kubernetes.io/projected/83455bdf-65dc-4f66-b943-55835e92b360-kube-api-access-l4kfc\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:37.879636 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:37.879620    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd2rw\" (UniqueName: \"kubernetes.io/projected/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-api-access-kd2rw\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:38.000316 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.000243    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:38.030934 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.030890    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/global-pull-secret-syncer-mmhd7" event={"ID":"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d","Type":"ContainerStarted","Data":"f4fb571e9cb3fe2a0f336911a73b85beae343bd4305c83c039786b15981b271e"}
Nov 02 17:29:38.094387 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.094361    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-network-diagnostics/network-check-target-nttxj"]
Nov 02 17:29:38.097798 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:38.097772    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda974c7e3_bf97_4575_bec1_c09fcd018f0d.slice/crio-15998cddba7811cb63969e4d4ccd9d484b4aa121c0a8c645977cfb696a73e5b5 WatchSource:0}: Error finding container 15998cddba7811cb63969e4d4ccd9d484b4aa121c0a8c645977cfb696a73e5b5: Status 404 returned error can't find the container with id 15998cddba7811cb63969e4d4ccd9d484b4aa121c0a8c645977cfb696a73e5b5
Nov 02 17:29:38.249228 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.249188    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:38.327870 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.327776    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/periodic-gathering-wphg6-wf57n"]
Nov 02 17:29:38.331423 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:38.331392    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83455bdf_65dc_4f66_b943_55835e92b360.slice/crio-00c2ae5d68441defef474549a48ed21d2249faeefb1b703c42569e621f3a57d2 WatchSource:0}: Error finding container 00c2ae5d68441defef474549a48ed21d2249faeefb1b703c42569e621f3a57d2: Status 404 returned error can't find the container with id 00c2ae5d68441defef474549a48ed21d2249faeefb1b703c42569e621f3a57d2
Nov 02 17:29:38.333487 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.333465    2542 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2"]
Nov 02 17:29:38.349863 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.349843    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2"
Nov 02 17:29:38.356651 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.356628    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:38.356751 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.356697    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:38.356751 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:38.356735    2542 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:38.356892 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:38.356801    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls podName:da9f1b66-0e1b-4c7e-a947-a897c00ccc19 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:39.356781465 +0000 UTC m=+36.131443705 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6bfcf" (UID: "da9f1b66-0e1b-4c7e-a947-a897c00ccc19") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:38.356892 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:38.356845    2542 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:29:38.356979 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:38.356912    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm podName:da9f1b66-0e1b-4c7e-a947-a897c00ccc19 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:39.356896256 +0000 UTC m=+36.131558505 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6bfcf" (UID: "da9f1b66-0e1b-4c7e-a947-a897c00ccc19") : configmap "kube-rbac-proxy" not found
Nov 02 17:29:38.369391 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.369363    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2"]
Nov 02 17:29:38.423947 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.423728    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:38.457139 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.457108    2542 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-kjmhh\" (UniqueName: \"kubernetes.io/projected/8064da0c-fc5d-493c-bcdf-4327716d4cc2-kube-api-access-kjmhh\") pod \"migrator-66df5d496b-lxxf2\" (UID: \"8064da0c-fc5d-493c-bcdf-4327716d4cc2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2"
Nov 02 17:29:38.462779 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.462747    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 17:29:38.462779 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.462747    2542 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vqf9q\""
Nov 02 17:29:38.462944 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.462766    2542 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 17:29:38.557730 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.557679    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjmhh\" (UniqueName: \"kubernetes.io/projected/8064da0c-fc5d-493c-bcdf-4327716d4cc2-kube-api-access-kjmhh\") pod \"migrator-66df5d496b-lxxf2\" (UID: \"8064da0c-fc5d-493c-bcdf-4327716d4cc2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2"
Nov 02 17:29:38.582901 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.582831    2542 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjmhh\" (UniqueName: \"kubernetes.io/projected/8064da0c-fc5d-493c-bcdf-4327716d4cc2-kube-api-access-kjmhh\") pod \"migrator-66df5d496b-lxxf2\" (UID: \"8064da0c-fc5d-493c-bcdf-4327716d4cc2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2"
Nov 02 17:29:38.617310 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.617270    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:38.672478 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:38.672450    2542 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2"
Nov 02 17:29:39.000236 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:39.000200    2542 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2"]
Nov 02 17:29:39.009895 ip-10-0-141-87 kubenswrapper[2542]: W1102 17:29:39.009863    2542 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8064da0c_fc5d_493c_bcdf_4327716d4cc2.slice/crio-c12ae599e6c930890621622a0cb873d074ecc075f38e97c1aba91f32c510fdd8 WatchSource:0}: Error finding container c12ae599e6c930890621622a0cb873d074ecc075f38e97c1aba91f32c510fdd8: Status 404 returned error can't find the container with id c12ae599e6c930890621622a0cb873d074ecc075f38e97c1aba91f32c510fdd8
Nov 02 17:29:39.035085 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:39.035048    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nttxj" event={"ID":"a974c7e3-bf97-4575-bec1-c09fcd018f0d","Type":"ContainerStarted","Data":"15998cddba7811cb63969e4d4ccd9d484b4aa121c0a8c645977cfb696a73e5b5"}
Nov 02 17:29:39.036750 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:39.036701    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wphg6-wf57n" event={"ID":"83455bdf-65dc-4f66-b943-55835e92b360","Type":"ContainerStarted","Data":"00c2ae5d68441defef474549a48ed21d2249faeefb1b703c42569e621f3a57d2"}
Nov 02 17:29:39.039598 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:39.039535    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2" event={"ID":"8064da0c-fc5d-493c-bcdf-4327716d4cc2","Type":"ContainerStarted","Data":"c12ae599e6c930890621622a0cb873d074ecc075f38e97c1aba91f32c510fdd8"}
Nov 02 17:29:39.330994 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:39.330907    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:39.365249 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:39.365205    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:39.365401 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:39.365272    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:39.365401 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:39.365363    2542 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:29:39.365497 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:39.365417    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm podName:da9f1b66-0e1b-4c7e-a947-a897c00ccc19 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:41.365400143 +0000 UTC m=+38.140062378 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6bfcf" (UID: "da9f1b66-0e1b-4c7e-a947-a897c00ccc19") : configmap "kube-rbac-proxy" not found
Nov 02 17:29:39.365773 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:39.365757    2542 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:39.365833 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:39.365803    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls podName:da9f1b66-0e1b-4c7e-a947-a897c00ccc19 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:41.365791662 +0000 UTC m=+38.140453897 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6bfcf" (UID: "da9f1b66-0e1b-4c7e-a947-a897c00ccc19") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:39.508287 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:39.508246    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:39.657920 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:39.657836    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:40.194284 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:40.194235    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:40.979726 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:40.979699    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:40.979884 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:40.979859    2542 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:29:40.979955 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:40.979923    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls podName:f59eba5d-a500-462e-9ecd-4ef9e553f873 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:48.979904205 +0000 UTC m=+45.754566453 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls") pod "dns-default-zmrdt" (UID: "f59eba5d-a500-462e-9ecd-4ef9e553f873") : secret "dns-default-metrics-tls" not found
Nov 02 17:29:41.047542 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:41.047488    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2" event={"ID":"8064da0c-fc5d-493c-bcdf-4327716d4cc2","Type":"ContainerStarted","Data":"0675abbd85b4030c862f933f71117ffb0019086a01fbec8168e2bf31f2755a25"}
Nov 02 17:29:41.090009 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:41.089956    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2" podStartSLOduration=1.2217508 podStartE2EDuration="3.089941345s" totalImagesPullingTime="1.868190545s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:38 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:41.089411085 +0000 UTC m=+37.864073342" watchObservedRunningTime="2026-11-02 17:29:41.089941345 +0000 UTC m=+37.864603601"
Nov 02 17:29:41.383218 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:41.383173    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:41.383662 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:41.383267    2542 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:41.383662 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:41.383307    2542 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:41.383662 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:41.383370    2542 configmap.go:193] Couldn't get configMap openshift-insights/kube-rbac-proxy: configmap "kube-rbac-proxy" not found
Nov 02 17:29:41.383662 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:41.383391    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls podName:da9f1b66-0e1b-4c7e-a947-a897c00ccc19 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:45.383369085 +0000 UTC m=+42.158031338 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6bfcf" (UID: "da9f1b66-0e1b-4c7e-a947-a897c00ccc19") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:41.383662 ip-10-0-141-87 kubenswrapper[2542]: E1102 17:29:41.383415    2542 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm podName:da9f1b66-0e1b-4c7e-a947-a897c00ccc19 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:45.38340425 +0000 UTC m=+42.158066520 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "kube-rbac-proxy-cm" (UniqueName: "kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm") pod "insights-runtime-extractor-6bfcf" (UID: "da9f1b66-0e1b-4c7e-a947-a897c00ccc19") : configmap "kube-rbac-proxy" not found
Nov 02 17:29:42.052353 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:42.052321    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wphg6-wf57n" event={"ID":"83455bdf-65dc-4f66-b943-55835e92b360","Type":"ContainerStarted","Data":"28db33cefa834b81d5b885b6a0ec776bb6e5248b0d66361db777a80d79851e25"}
Nov 02 17:29:42.124650 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:42.124607    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-insights/periodic-gathering-wphg6-wf57n" podStartSLOduration=2.327580588 podStartE2EDuration="5.124593063s" totalImagesPullingTime="2.797012475s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:37 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:42.123994705 +0000 UTC m=+38.898656963" watchObservedRunningTime="2026-11-02 17:29:42.124593063 +0000 UTC m=+38.899255321"
Nov 02 17:29:42.983320 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:42.983285    2542 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:43.056392 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:43.056352    2542 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-network-diagnostics/network-check-target-nttxj" event={"ID":"a974c7e3-bf97-4575-bec1-c09fcd018f0d","Type":"ContainerStarted","Data":"db4c79e1e12f84474d1de0d33eb12cd55e7ad6738e4381205d1507842bf29bf0"}
Nov 02 17:29:43.104120 ip-10-0-141-87 kubenswrapper[2542]: I1102 17:29:43.104069    2542 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-network-diagnostics/network-check-target-nttxj" podStartSLOduration=36.161816095 podStartE2EDuration="40.104056525s" totalImagesPullingTime="3.94224043s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:29:03 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:29:43.103518775 +0000 UTC m=+39.878181028" watchObservedRunningTime="2026-11-02 17:29:43.104056525 +0000 UTC m=+39.878718806"
Nov 02 17:29:43.459799 ip-10-0-141-87 systemd[1]: Stopping Kubernetes Kubelet...
Nov 02 17:29:43.470158 ip-10-0-141-87 systemd[1]: kubelet.service: Deactivated successfully.
Nov 02 17:29:43.470328 ip-10-0-141-87 systemd[1]: Stopped Kubernetes Kubelet.
Nov 02 17:29:43.470374 ip-10-0-141-87 systemd[1]: kubelet.service: Consumed 2.329s CPU time.
Nov 02 17:29:43.479313 ip-10-0-141-87 systemd[1]: Starting Kubernetes Kubelet...
Nov 02 17:29:43.537882 ip-10-0-141-87 kubenswrapper[4456]: Flag --container-runtime-endpoint has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:29:43.537882 ip-10-0-141-87 kubenswrapper[4456]: Flag --minimum-container-ttl-duration has been deprecated, Use --eviction-hard or --eviction-soft instead. Will be removed in a future version.
Nov 02 17:29:43.537882 ip-10-0-141-87 kubenswrapper[4456]: Flag --volume-plugin-dir has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:29:43.537882 ip-10-0-141-87 kubenswrapper[4456]: Flag --system-reserved has been deprecated, This parameter should be set via the config file specified by the Kubelet's --config flag. See https://kubernetes.io/docs/tasks/administer-cluster/kubelet-config-file/ for more information.
Nov 02 17:29:43.539907 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539832    4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:43.539907 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539906    4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:43.539907 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539910    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539913    4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539915    4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539918    4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539921    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539923    4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539926    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539931    4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539936    4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539940    4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539943    4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539946    4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539949    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539952    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539954    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539957    4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539960    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539962    4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539972    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:43.540002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539975    4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539978    4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539980    4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539983    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539986    4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539989    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539991    4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539994    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539996    4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.539999    4456 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540001    4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540004    4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540007    4456 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540010    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540013    4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540016    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540019    4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540021    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540024    4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:43.540617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540026    4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540029    4456 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540031    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540034    4456 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540036    4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540039    4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540041    4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540044    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540046    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540049    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540051    4456 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540054    4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540056    4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540059    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540063    4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540066    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540068    4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540071    4456 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540073    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540076    4456 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:43.541120 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540080    4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540083    4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540085    4456 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540088    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540090    4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540093    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540095    4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540097    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540100    4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540102    4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540105    4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540108    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540111    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540113    4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540116    4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540118    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540120    4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540123    4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540125    4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540127    4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:43.541605 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540130    4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540132    4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540135    4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540137    4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540140    4456 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540142    4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540144    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540147    4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540150    4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540152    4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540155    4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540157    4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540160    4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540162    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540165    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540168    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540170    4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540174    4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540177    4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540179    4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:43.542118 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540182    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540184    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540186    4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540189    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540191    4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540195    4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.540996    4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541004    4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541009    4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541012    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541015    4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541018    4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541021    4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541024    4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541027    4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541029    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541032    4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541034    4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541037    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:43.542661 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541039    4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541042    4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541044    4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541047    4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541049    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541052    4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541055    4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541057    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541060    4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541063    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541065    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541068    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541070    4456 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541073    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541075    4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541077    4456 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541080    4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541083    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541085    4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:43.543153 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541093    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541096    4456 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541098    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541101    4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541105    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541107    4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541109    4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541113    4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541117    4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541120    4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541123    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541126    4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541129    4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541131    4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541134    4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541137    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541139    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541142    4456 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541145    4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:43.543631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541148    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541150    4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541153    4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541155    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541158    4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541160    4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541163    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541165    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541168    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541170    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541173    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541175    4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541178    4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541180    4456 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541183    4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541186    4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541188    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541191    4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541194    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541197    4456 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:43.544150 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541199    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541202    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541205    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541207    4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541211    4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541213    4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541216    4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541218    4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541221    4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541223    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541230    4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541232    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541235    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541237    4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541240    4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541243    4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541245    4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541249    4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541253    4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:43.544643 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541257    4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541262    4456 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541265    4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541278    4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541283    4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541286    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541289    4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541293    4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541296    4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541299    4456 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541303    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541305    4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541309    4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541312    4456 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541314    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541317    4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541836    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541847    4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541850    4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:43.545132 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541854    4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541856    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541859    4456 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541863    4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541869    4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541873    4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541875    4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541878    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541880    4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541883    4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541885    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541888    4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541890    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541893    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541895    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541898    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541900    4456 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541903    4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541905    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541908    4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:43.545601 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541910    4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541921    4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541924    4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541927    4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541929    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541932    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541934    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541937    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541940    4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541942    4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541945    4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541947    4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541950    4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541952    4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541955    4456 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541957    4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541961    4456 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541964    4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541966    4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541969    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:43.546122 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541971    4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541974    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541976    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541979    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541981    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541984    4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541986    4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541988    4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541991    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541994    4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541996    4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.541999    4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542001    4456 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542004    4456 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542008    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542011    4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542013    4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542015    4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542018    4456 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542021    4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:43.546920 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542023    4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542025    4456 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542028    4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542031    4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542033    4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542036    4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542038    4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542041    4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542044    4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542047    4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542049    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542052    4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542054    4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542057    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542060    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542064    4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542067    4456 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542070    4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542072    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:43.547482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542075    4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542077    4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542080    4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542082    4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542085    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542087    4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542090    4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542093    4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542096    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542099    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542101    4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542104    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542107    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542109    4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542112    4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542115    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542118    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542120    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542123    4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:43.548027 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542125    4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542128    4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542131    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542134    4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542136    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542226    4456 flags.go:64] FLAG: --address="0.0.0.0"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542233    4456 flags.go:64] FLAG: --allowed-unsafe-sysctls="[]"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542240    4456 flags.go:64] FLAG: --anonymous-auth="true"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542244    4456 flags.go:64] FLAG: --application-metrics-count-limit="100"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542256    4456 flags.go:64] FLAG: --authentication-token-webhook="false"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542259    4456 flags.go:64] FLAG: --authentication-token-webhook-cache-ttl="2m0s"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542263    4456 flags.go:64] FLAG: --authorization-mode="AlwaysAllow"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542267    4456 flags.go:64] FLAG: --authorization-webhook-cache-authorized-ttl="5m0s"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542270    4456 flags.go:64] FLAG: --authorization-webhook-cache-unauthorized-ttl="30s"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542273    4456 flags.go:64] FLAG: --boot-id-file="/proc/sys/kernel/random/boot_id"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542277    4456 flags.go:64] FLAG: --bootstrap-kubeconfig="/etc/kubernetes/kubeconfig"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542280    4456 flags.go:64] FLAG: --cert-dir="/var/lib/kubelet/pki"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542283    4456 flags.go:64] FLAG: --cgroup-driver="cgroupfs"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542286    4456 flags.go:64] FLAG: --cgroup-root=""
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542288    4456 flags.go:64] FLAG: --cgroups-per-qos="true"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542292    4456 flags.go:64] FLAG: --client-ca-file=""
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542295    4456 flags.go:64] FLAG: --cloud-provider="external"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542301    4456 flags.go:64] FLAG: --cluster-dns="[]"
Nov 02 17:29:43.548490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542305    4456 flags.go:64] FLAG: --cluster-domain=""
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542308    4456 flags.go:64] FLAG: --config="/etc/kubernetes/kubelet.conf"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542311    4456 flags.go:64] FLAG: --config-dir="/etc/openshift/kubelet.conf.d"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542314    4456 flags.go:64] FLAG: --container-hints="/etc/cadvisor/container_hints.json"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542317    4456 flags.go:64] FLAG: --container-log-max-files="5"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542321    4456 flags.go:64] FLAG: --container-log-max-size="10Mi"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542324    4456 flags.go:64] FLAG: --container-runtime-endpoint="/var/run/crio/crio.sock"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542327    4456 flags.go:64] FLAG: --containerd="/run/containerd/containerd.sock"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542329    4456 flags.go:64] FLAG: --containerd-namespace="k8s.io"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542333    4456 flags.go:64] FLAG: --contention-profiling="false"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542336    4456 flags.go:64] FLAG: --cpu-cfs-quota="true"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542339    4456 flags.go:64] FLAG: --cpu-cfs-quota-period="100ms"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542343    4456 flags.go:64] FLAG: --cpu-manager-policy="none"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542345    4456 flags.go:64] FLAG: --cpu-manager-policy-options=""
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542349    4456 flags.go:64] FLAG: --cpu-manager-reconcile-period="10s"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542353    4456 flags.go:64] FLAG: --enable-controller-attach-detach="true"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542356    4456 flags.go:64] FLAG: --enable-debugging-handlers="true"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542359    4456 flags.go:64] FLAG: --enable-load-reader="false"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542362    4456 flags.go:64] FLAG: --enable-server="true"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542365    4456 flags.go:64] FLAG: --enforce-node-allocatable="[pods]"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542370    4456 flags.go:64] FLAG: --event-burst="100"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542373    4456 flags.go:64] FLAG: --event-qps="50"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542376    4456 flags.go:64] FLAG: --event-storage-age-limit="default=0"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542379    4456 flags.go:64] FLAG: --event-storage-event-limit="default=0"
Nov 02 17:29:43.549189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542382    4456 flags.go:64] FLAG: --eviction-hard=""
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542386    4456 flags.go:64] FLAG: --eviction-max-pod-grace-period="0"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542389    4456 flags.go:64] FLAG: --eviction-minimum-reclaim=""
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542392    4456 flags.go:64] FLAG: --eviction-pressure-transition-period="5m0s"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542395    4456 flags.go:64] FLAG: --eviction-soft=""
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542398    4456 flags.go:64] FLAG: --eviction-soft-grace-period=""
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542401    4456 flags.go:64] FLAG: --exit-on-lock-contention="false"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542404    4456 flags.go:64] FLAG: --experimental-allocatable-ignore-eviction="false"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542407    4456 flags.go:64] FLAG: --experimental-mounter-path=""
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542410    4456 flags.go:64] FLAG: --fail-cgroupv1="true"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542413    4456 flags.go:64] FLAG: --fail-swap-on="true"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542416    4456 flags.go:64] FLAG: --feature-gates=""
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542423    4456 flags.go:64] FLAG: --file-check-frequency="20s"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542426    4456 flags.go:64] FLAG: --global-housekeeping-interval="1m0s"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542429    4456 flags.go:64] FLAG: --hairpin-mode="promiscuous-bridge"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542432    4456 flags.go:64] FLAG: --healthz-bind-address="127.0.0.1"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542435    4456 flags.go:64] FLAG: --healthz-port="10248"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542438    4456 flags.go:64] FLAG: --help="false"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542441    4456 flags.go:64] FLAG: --hostname-override="ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542444    4456 flags.go:64] FLAG: --housekeeping-interval="10s"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542446    4456 flags.go:64] FLAG: --http-check-frequency="20s"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542449    4456 flags.go:64] FLAG: --image-credential-provider-bin-dir="/usr/libexec/kubelet-image-credential-provider-plugins"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542452    4456 flags.go:64] FLAG: --image-credential-provider-config="/etc/kubernetes/credential-providers/ecr-credential-provider.yaml"
Nov 02 17:29:43.549805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542455    4456 flags.go:64] FLAG: --image-gc-high-threshold="85"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542458    4456 flags.go:64] FLAG: --image-gc-low-threshold="80"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542461    4456 flags.go:64] FLAG: --image-service-endpoint=""
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542464    4456 flags.go:64] FLAG: --kernel-memcg-notification="false"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542466    4456 flags.go:64] FLAG: --kube-api-burst="100"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542469    4456 flags.go:64] FLAG: --kube-api-content-type="application/vnd.kubernetes.protobuf"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542472    4456 flags.go:64] FLAG: --kube-api-qps="50"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542475    4456 flags.go:64] FLAG: --kube-reserved=""
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542478    4456 flags.go:64] FLAG: --kube-reserved-cgroup=""
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542480    4456 flags.go:64] FLAG: --kubeconfig="/var/lib/kubelet/kubeconfig"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542483    4456 flags.go:64] FLAG: --kubelet-cgroups=""
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542486    4456 flags.go:64] FLAG: --local-storage-capacity-isolation="true"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542489    4456 flags.go:64] FLAG: --lock-file=""
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542491    4456 flags.go:64] FLAG: --log-cadvisor-usage="false"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542495    4456 flags.go:64] FLAG: --log-flush-frequency="5s"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542498    4456 flags.go:64] FLAG: --log-json-info-buffer-size="0"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542503    4456 flags.go:64] FLAG: --log-json-split-stream="false"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542506    4456 flags.go:64] FLAG: --log-text-info-buffer-size="0"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542509    4456 flags.go:64] FLAG: --log-text-split-stream="false"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542512    4456 flags.go:64] FLAG: --logging-format="text"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542515    4456 flags.go:64] FLAG: --machine-id-file="/etc/machine-id,/var/lib/dbus/machine-id"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542519    4456 flags.go:64] FLAG: --make-iptables-util-chains="true"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542522    4456 flags.go:64] FLAG: --manifest-url=""
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542524    4456 flags.go:64] FLAG: --manifest-url-header=""
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542529    4456 flags.go:64] FLAG: --max-housekeeping-interval="15s"
Nov 02 17:29:43.550406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542532    4456 flags.go:64] FLAG: --max-open-files="1000000"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542536    4456 flags.go:64] FLAG: --max-pods="110"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542539    4456 flags.go:64] FLAG: --maximum-dead-containers="-1"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542541    4456 flags.go:64] FLAG: --maximum-dead-containers-per-container="1"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542544    4456 flags.go:64] FLAG: --memory-manager-policy="None"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542547    4456 flags.go:64] FLAG: --minimum-container-ttl-duration="6m0s"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542550    4456 flags.go:64] FLAG: --minimum-image-ttl-duration="2m0s"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542553    4456 flags.go:64] FLAG: --node-ip="0.0.0.0"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542556    4456 flags.go:64] FLAG: --node-labels="node-role.kubernetes.io/worker=,node.openshift.io/os_id=rhel"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542563    4456 flags.go:64] FLAG: --node-status-max-images="50"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542566    4456 flags.go:64] FLAG: --node-status-update-frequency="10s"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542569    4456 flags.go:64] FLAG: --oom-score-adj="-999"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542572    4456 flags.go:64] FLAG: --pod-cidr=""
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542574    4456 flags.go:64] FLAG: --pod-manifest-path=""
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542577    4456 flags.go:64] FLAG: --pod-max-pids="-1"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542580    4456 flags.go:64] FLAG: --pods-per-core="0"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542583    4456 flags.go:64] FLAG: --port="10250"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542586    4456 flags.go:64] FLAG: --protect-kernel-defaults="false"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542588    4456 flags.go:64] FLAG: --provider-id="aws:///us-east-1a/i-086bc04b1e25d38e7"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542591    4456 flags.go:64] FLAG: --qos-reserved=""
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542594    4456 flags.go:64] FLAG: --read-only-port="10255"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542597    4456 flags.go:64] FLAG: --register-node="true"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542600    4456 flags.go:64] FLAG: --register-with-taints=""
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542605    4456 flags.go:64] FLAG: --registry-burst="10"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542608    4456 flags.go:64] FLAG: --registry-qps="5"
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542610    4456 flags.go:64] FLAG: --reserved-cpus=""
Nov 02 17:29:43.551020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542614    4456 flags.go:64] FLAG: --reserved-memory=""
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542617    4456 flags.go:64] FLAG: --resolv-conf="/etc/resolv.conf"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542620    4456 flags.go:64] FLAG: --root-dir="/var/lib/kubelet"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542624    4456 flags.go:64] FLAG: --rotate-certificates="false"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542626    4456 flags.go:64] FLAG: --rotate-server-certificates="false"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542629    4456 flags.go:64] FLAG: --runonce="false"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542633    4456 flags.go:64] FLAG: --runtime-cgroups="/system.slice/crio.service"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542636    4456 flags.go:64] FLAG: --runtime-request-timeout="2m0s"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542639    4456 flags.go:64] FLAG: --seccomp-default="false"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542642    4456 flags.go:64] FLAG: --serialize-image-pulls="true"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542645    4456 flags.go:64] FLAG: --storage-driver-buffer-duration="1m0s"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542648    4456 flags.go:64] FLAG: --storage-driver-db="cadvisor"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542651    4456 flags.go:64] FLAG: --storage-driver-host="localhost:8086"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542654    4456 flags.go:64] FLAG: --storage-driver-password="root"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542673    4456 flags.go:64] FLAG: --storage-driver-secure="false"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542677    4456 flags.go:64] FLAG: --storage-driver-table="stats"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542680    4456 flags.go:64] FLAG: --storage-driver-user="root"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542683    4456 flags.go:64] FLAG: --streaming-connection-idle-timeout="4h0m0s"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542686    4456 flags.go:64] FLAG: --sync-frequency="1m0s"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542690    4456 flags.go:64] FLAG: --system-cgroups=""
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542692    4456 flags.go:64] FLAG: --system-reserved="cpu=500m,ephemeral-storage=1Gi,memory=1Gi"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542697    4456 flags.go:64] FLAG: --system-reserved-cgroup=""
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542700    4456 flags.go:64] FLAG: --tls-cert-file=""
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542703    4456 flags.go:64] FLAG: --tls-cipher-suites="[]"
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542707    4456 flags.go:64] FLAG: --tls-min-version=""
Nov 02 17:29:43.551722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542710    4456 flags.go:64] FLAG: --tls-private-key-file=""
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542713    4456 flags.go:64] FLAG: --topology-manager-policy="none"
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542716    4456 flags.go:64] FLAG: --topology-manager-policy-options=""
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542719    4456 flags.go:64] FLAG: --topology-manager-scope="container"
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542723    4456 flags.go:64] FLAG: --v="2"
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542727    4456 flags.go:64] FLAG: --version="false"
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542731    4456 flags.go:64] FLAG: --vmodule=""
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542735    4456 flags.go:64] FLAG: --volume-plugin-dir="/etc/kubernetes/kubelet-plugins/volume/exec"
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.542739    4456 flags.go:64] FLAG: --volume-stats-agg-period="1m0s"
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542897    4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542902    4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542905    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542908    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542915    4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542918    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542921    4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542925    4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542929    4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542932    4456 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542934    4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542937    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:43.552444 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542940    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542942    4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542945    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542947    4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542950    4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542952    4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542955    4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542957    4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542960    4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542962    4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542964    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542967    4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542970    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542972    4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542975    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542978    4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542981    4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542983    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542986    4456 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:43.553014 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542988    4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542991    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542994    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542996    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.542999    4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543003    4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543005    4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543007    4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543010    4456 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543013    4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543015    4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543018    4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543020    4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543022    4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543025    4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543027    4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543030    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543032    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543034    4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543037    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:43.553631 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543040    4456 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543043    4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543045    4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543047    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543050    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543052    4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543055    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543057    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543061    4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543064    4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543067    4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543071    4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543074    4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543077    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543080    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543083    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543085    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543089    4456 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543092    4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:43.554510 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543094    4456 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543096    4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543099    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543101    4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543104    4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543106    4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543108    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543111    4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543113    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543116    4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543118    4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543120    4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543123    4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543125    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543128    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543131    4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543133    4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543136    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543138    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:43.555111 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543140    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543143    4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543146    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543149    4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543152    4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543154    4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543157    4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543160    4456 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543163    4456 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543166    4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543168    4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543172    4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543174    4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543177    4456 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543179    4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543182    4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:43.555587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543184    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.543210    4456 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543245    4456 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543250    4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543253    4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543256    4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543258    4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543261    4456 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543263    4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543266    4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543269    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543271    4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543274    4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543276    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543279    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543281    4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:43.556087 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543284    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543287    4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543291    4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543296    4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543300    4456 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543302    4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543304    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543307    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543309    4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543312    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543315    4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543319    4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543322    4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543325    4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543327    4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543330    4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543332    4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543334    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543337    4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:43.556502 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543339    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543342    4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543344    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543347    4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543349    4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543352    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543354    4456 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543357    4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543359    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543361    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543364    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543366    4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543369    4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543371    4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543374    4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543377    4456 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543380    4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543383    4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543385    4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543388    4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:43.556993 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543390    4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543393    4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543395    4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543398    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543402    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543405    4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543407    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543410    4456 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543412    4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543415    4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543417    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543419    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543422    4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543424    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543427    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543429    4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543431    4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543435    4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543439    4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:43.557482 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543442    4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543444    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543447    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543449    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543452    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543455    4456 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543457    4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543460    4456 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543462    4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543466    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543469    4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543471    4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543474    4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543476    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543478    4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543481    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543483    4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543487    4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543490    4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:43.558085 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543492    4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543495    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543497    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543500    4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543502    4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543504    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543507    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543509    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543512    4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543514    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543517    4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543519    4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543521    4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543524    4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.543527    4456 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.548314    4456 server.go:525] "Kubelet version" kubeletVersion="v1.35.9"
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.548330    4456 server.go:527] "Golang settings" GOGC="" GOMAXPROCS="" GOTRACEBACK=""
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.548349    4456 watchdog_linux.go:95] "Systemd watchdog is not enabled"
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.548353    4456 watchdog_linux.go:138] "Systemd watchdog is not enabled or the interval is invalid, so health checking will not be started."
Nov 02 17:29:43.558551 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548411    4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548416    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548419    4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548423    4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548425    4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548430    4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548434    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548437    4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548440    4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548442    4456 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548445    4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548448    4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548451    4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548453    4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548455    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548458    4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548461    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548463    4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548466    4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:43.559018 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548468    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548471    4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548473    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548476    4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548478    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548481    4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548483    4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548486    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548492    4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548495    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548498    4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548500    4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548503    4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548506    4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548508    4456 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548511    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548513    4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548516    4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548518    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:43.559477 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548521    4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548523    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548526    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548529    4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548531    4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548534    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548536    4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548539    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548543    4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548547    4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548550    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548553    4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548556    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548558    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548560    4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548578    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548582    4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548586    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548588    4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:43.560115 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548591    4456 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548594    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548597    4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548599    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548602    4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548604    4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548607    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548609    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548612    4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548614    4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548617    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548620    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548622    4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548625    4456 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548627    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548630    4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548633    4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548635    4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548638    4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548640    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:43.560576 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548643    4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548645    4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548648    4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548651    4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548654    4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548656    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548659    4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548661    4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548663    4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548666    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548668    4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548671    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548673    4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548676    4456 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548678    4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548680    4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548683    4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548685    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548688    4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548690    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:43.561191 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548693    4456 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548695    4456 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548698    4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548700    4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548703    4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548705    4456 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548708    4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548710    4456 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548713    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.548740    4456 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548890    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548896    4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548899    4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548902    4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548904    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548907    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:43.561871 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548910    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548913    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548915    4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548917    4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548920    4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548922    4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548925    4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548927    4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548930    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548932    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548935    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548937    4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548939    4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548942    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548944    4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548947    4456 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548950    4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548953    4456 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548958    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548961    4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:43.562378 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548963    4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548966    4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548968    4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548971    4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548973    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548976    4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548978    4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548981    4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548983    4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548986    4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548988    4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.548992    4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549001    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549004    4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549006    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549009    4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549011    4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549014    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549016    4456 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549019    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:43.563092 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549021    4456 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549024    4456 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549026    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549028    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549031    4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549033    4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549036    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549038    4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549041    4456 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549044    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549048    4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549050    4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549053    4456 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549056    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549058    4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549061    4456 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549063    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549066    4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549069    4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549071    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:43.563826 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549073    4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549076    4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549078    4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549081    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549083    4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549086    4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549089    4456 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549092    4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549094    4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549097    4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549100    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549102    4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549104    4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549107    4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549109    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549112    4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549114    4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549117    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549119    4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:43.564428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549122    4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549124    4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549128    4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549134    4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549137    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549141    4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549145    4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549148    4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549151    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549154    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549156    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549159    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549161    4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549164    4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549166    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549169    4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549171    4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549174    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549176    4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:43.564964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549179    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549182    4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.549206    4456 feature_gate.go:465] feature gates: {map[CBORServingAndStorage:false ClientsAllowCBOR:false ClientsPreferCBOR:false DRAPartitionableDevices:false EventedPLEG:false KMSv1:true MaxUnavailableStatefulSet:false MinimumKubeletVersion:false MutableCSINodeAllocatableCount:true MutatingAdmissionPolicy:false RouteExternalCertificate:true SELinuxMount:false ServiceAccountTokenNodeBinding:true StoragePerformantSecurityPolicy:true UserNamespacesSupport:true]}
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549239    4456 feature_gate.go:386] unrecognized feature gate: CRIOCredentialProviderConfig
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549243    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAzure
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549246    4456 feature_gate.go:386] unrecognized feature gate: GCPSovereignCloudInstall
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549249    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagement
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549252    4456 feature_gate.go:386] unrecognized feature gate: NetworkConnect
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549254    4456 feature_gate.go:386] unrecognized feature gate: BootImageSkewEnforcement
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549257    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNS
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549260    4456 feature_gate.go:386] unrecognized feature gate: OVNObservability
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549262    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationOpenStack
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549264    4456 feature_gate.go:386] unrecognized feature gate: Example
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549267    4456 feature_gate.go:386] unrecognized feature gate: SignatureStores
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549270    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDC
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549272    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementAWS
Nov 02 17:29:43.565525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549275    4456 feature_gate.go:386] unrecognized feature gate: AWSServiceLBNetworkSecurityGroup
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549294    4456 feature_gate.go:386] unrecognized feature gate: AzureMultiDisk
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549298    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiDisk
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549302    4456 feature_gate.go:386] unrecognized feature gate: DyanmicServiceEndpointIBMCloud
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549305    4456 feature_gate.go:386] unrecognized feature gate: OSStreams
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549308    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementOpenStack
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549311    4456 feature_gate.go:386] unrecognized feature gate: NewOLMConfigAPI
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549314    4456 feature_gate.go:386] unrecognized feature gate: AWSDedicatedHosts
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549316    4456 feature_gate.go:386] unrecognized feature gate: DualReplica
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549319    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstall
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549321    4456 feature_gate.go:386] unrecognized feature gate: GatewayAPIWithoutOLM
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549324    4456 feature_gate.go:386] unrecognized feature gate: Example2
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549326    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementGCP
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549328    4456 feature_gate.go:386] unrecognized feature gate: IrreconcilableMachineConfig
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549331    4456 feature_gate.go:386] unrecognized feature gate: InsightsConfig
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549333    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUIDAndExtraClaimMappings
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549336    4456 feature_gate.go:386] unrecognized feature gate: NewOLMCatalogdAPIV1Metas
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549339    4456 feature_gate.go:386] unrecognized feature gate: OpenShiftPodSecurityAdmission
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549341    4456 feature_gate.go:386] unrecognized feature gate: NewOLMOwnSingleNamespace
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549343    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdatePreflight
Nov 02 17:29:43.566086 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549346    4456 feature_gate.go:386] unrecognized feature gate: ConfidentialCluster
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549348    4456 feature_gate.go:386] unrecognized feature gate: TLSAdherence
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549351    4456 feature_gate.go:386] unrecognized feature gate: EventTTL
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549353    4456 feature_gate.go:386] unrecognized feature gate: VSphereConfigurableMaxAllowedBlockVolumesPerNode
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549356    4456 feature_gate.go:386] unrecognized feature gate: IngressControllerDynamicConfigurationManager
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549358    4456 feature_gate.go:386] unrecognized feature gate: ImageModeStatusReporting
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549360    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCExternalClaimsSourcing
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549363    4456 feature_gate.go:386] unrecognized feature gate: AdditionalStorageConfig
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549365    4456 feature_gate.go:386] unrecognized feature gate: EtcdBackendQuota
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549368    4456 feature_gate.go:386] unrecognized feature gate: ImageStreamImportMode
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549372    4456 feature_gate.go:386] unrecognized feature gate: ExternalOIDCWithUpstreamParity
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549374    4456 feature_gate.go:386] unrecognized feature gate: AWSEuropeanSovereignCloudInstall
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549377    4456 feature_gate.go:386] unrecognized feature gate: MultiArchInstallAzure
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549379    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationAWS
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549382    4456 feature_gate.go:386] unrecognized feature gate: AzureClusterHostedDNSInstall
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549385    4456 feature_gate.go:386] unrecognized feature gate: ConfigurablePKI
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549387    4456 feature_gate.go:386] unrecognized feature gate: NoOverlayMode
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549391    4456 feature_gate.go:409] Setting GA feature gate ServiceAccountTokenNodeBinding=true. It will be removed in a future release.
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549396    4456 feature_gate.go:386] unrecognized feature gate: ProvisioningRequestAvailable
Nov 02 17:29:43.566941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549399    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigration
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549403    4456 feature_gate.go:407] Setting deprecated feature gate KMSv1=true. It will be removed in a future release.
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549406    4456 feature_gate.go:386] unrecognized feature gate: DNSNameResolver
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549409    4456 feature_gate.go:386] unrecognized feature gate: UpgradeStatus
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549411    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementVSphere
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549414    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryption
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549417    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIComputeInstall
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549419    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerificationPKI
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549422    4456 feature_gate.go:386] unrecognized feature gate: BuildCSIVolumes
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549424    4456 feature_gate.go:386] unrecognized feature gate: AWSDualStackInstall
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549426    4456 feature_gate.go:386] unrecognized feature gate: ClusterUpdateAcceptRisks
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549430    4456 feature_gate.go:386] unrecognized feature gate: ConsolePluginContentSecurityPolicy
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549432    4456 feature_gate.go:386] unrecognized feature gate: GCPDualStackInstall
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549434    4456 feature_gate.go:386] unrecognized feature gate: AWSClusterHostedDNSInstall
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549437    4456 feature_gate.go:386] unrecognized feature gate: NoRegistryClusterInstall
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549439    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementPowerVS
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549442    4456 feature_gate.go:386] unrecognized feature gate: NewOLMWebhookProviderOpenshiftServiceCA
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549444    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIOperatorDisableMachineHealthCheckController
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549447    4456 feature_gate.go:386] unrecognized feature gate: ShortCertRotation
Nov 02 17:29:43.567475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549449    4456 feature_gate.go:386] unrecognized feature gate: BootcNodeManagement
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549451    4456 feature_gate.go:386] unrecognized feature gate: AzureDedicatedHosts
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549454    4456 feature_gate.go:386] unrecognized feature gate: MachineAPIMigrationVSphere
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549456    4456 feature_gate.go:386] unrecognized feature gate: KMSEncryptionProvider
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549460    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIMachineManagementBareMetal
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549462    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiVCenterDay2
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549466    4456 feature_gate.go:386] unrecognized feature gate: HyperShiftOnlyDynamicResourceAllocation
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549470    4456 feature_gate.go:386] unrecognized feature gate: OnPremDNSRecords
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549474    4456 feature_gate.go:386] unrecognized feature gate: ExternalSnapshotMetadata
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549478    4456 feature_gate.go:386] unrecognized feature gate: VolumeGroupSnapshot
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549482    4456 feature_gate.go:386] unrecognized feature gate: VSphereHostVMGroupZonal
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549485    4456 feature_gate.go:386] unrecognized feature gate: ClusterVersionOperatorConfiguration
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549487    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpointsInstall
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549490    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIControlPlaneInstall
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549492    4456 feature_gate.go:386] unrecognized feature gate: ClusterAPIInstallIBMCloud
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549495    4456 feature_gate.go:386] unrecognized feature gate: NewOLMBoxCutterRuntime
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549497    4456 feature_gate.go:386] unrecognized feature gate: MixedCPUsAllocation
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549499    4456 feature_gate.go:386] unrecognized feature gate: VSphereMultiNetworks
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549502    4456 feature_gate.go:386] unrecognized feature gate: AzureWorkloadIdentity
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549504    4456 feature_gate.go:386] unrecognized feature gate: EVPN
Nov 02 17:29:43.568005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549507    4456 feature_gate.go:386] unrecognized feature gate: MetricsCollectionProfiles
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549510    4456 feature_gate.go:386] unrecognized feature gate: NewOLM
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549512    4456 feature_gate.go:386] unrecognized feature gate: GCPCustomAPIEndpoints
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549514    4456 feature_gate.go:386] unrecognized feature gate: AzureDualStackInstall
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549517    4456 feature_gate.go:386] unrecognized feature gate: MultiDiskSetup
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549520    4456 feature_gate.go:386] unrecognized feature gate: CRDCompatibilityRequirementOperator
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549523    4456 feature_gate.go:386] unrecognized feature gate: VSphereMixedNodeEnv
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549525    4456 feature_gate.go:386] unrecognized feature gate: NutanixMultiSubnets
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549528    4456 feature_gate.go:386] unrecognized feature gate: SigstoreImageVerification
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549530    4456 feature_gate.go:386] unrecognized feature gate: AutomatedEtcdBackup
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549533    4456 feature_gate.go:386] unrecognized feature gate: ManagedBootImagesCPMS
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549536    4456 feature_gate.go:386] unrecognized feature gate: NewOLMPreflightPermissionChecks
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549539    4456 feature_gate.go:386] unrecognized feature gate: InsightsOnDemandDataGather
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549541    4456 feature_gate.go:386] unrecognized feature gate: UserNamespacesPodSecurityStandards
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:43.549544    4456 feature_gate.go:386] unrecognized feature gate: ClusterMonitoringConfig
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.549600    4456 server.go:957] "Client rotation is on, will bootstrap in background"
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.550583    4456 bootstrap.go:85] "Current kubeconfig file contents are still valid, no bootstrap necessary"
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.550638    4456 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-client-current.pem"
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.550838    4456 server.go:1014] "Starting client certificate rotation"
Nov 02 17:29:43.568492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.550928    4456 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kube-apiserver-client-kubelet"
Nov 02 17:29:43.568992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.550975    4456 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kube-apiserver-client-kubelet" expiration="2028-11-01 17:24:03 +0000 UTC" deadline="2028-06-05 21:14:21.64182462 +0000 UTC"
Nov 02 17:29:43.568992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.551003    4456 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kube-apiserver-client-kubelet" sleep="13947h44m38.090823864s"
Nov 02 17:29:43.568992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.551287    4456 dynamic_cafile_content.go:123] "Loaded a new CA Bundle and Verifier" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:29:43.568992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.552380    4456 dynamic_cafile_content.go:161] "Starting controller" name="client-ca-bundle::/etc/kubernetes/kubelet-ca.crt"
Nov 02 17:29:43.568992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.553792    4456 log.go:25] "Validated CRI v1 runtime API"
Nov 02 17:29:43.568992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.555339    4456 log.go:25] "Validated CRI v1 image API"
Nov 02 17:29:43.568992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.556062    4456 server.go:1424] "Using cgroup driver setting received from the CRI runtime" cgroupDriver="systemd"
Nov 02 17:29:43.568992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.557540    4456 fs.go:135] Filesystem UUIDs: map[2577969f-b148-4472-84ae-e5ed82a01786:/dev/nvme0n1p3 6ee8195b-f86e-4e62-94df-29f84cdf6339:/dev/nvme0n1p4 7B77-95E7:/dev/nvme0n1p2]
Nov 02 17:29:43.569224 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.557555    4456 fs.go:136] Filesystem partitions: map[/dev/nvme0n1p3:{mountpoint:/boot major:259 minor:3 fsType:ext4 blockSize:0} /dev/nvme0n1p4:{mountpoint:/var major:259 minor:4 fsType:xfs blockSize:0} /dev/shm:{mountpoint:/dev/shm major:0 minor:24 fsType:tmpfs blockSize:0} /run:{mountpoint:/run major:0 minor:26 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/00c2ae5d68441defef474549a48ed21d2249faeefb1b703c42569e621f3a57d2/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/00c2ae5d68441defef474549a48ed21d2249faeefb1b703c42569e621f3a57d2/userdata/shm major:0 minor:222 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/0e45823f1f3e78e26f6c3e2889edb7592f7debd8b8dbcc4dcf99f8f291f90da1/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/0e45823f1f3e78e26f6c3e2889edb7592f7debd8b8dbcc4dcf99f8f291f90da1/userdata/shm major:0 minor:84 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/15998cddba7811cb63969e4d4ccd9d484b4aa121c0a8c645977cfb696a73e5b5/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/15998cddba7811cb63969e4d4ccd9d484b4aa121c0a8c645977cfb696a73e5b5/userdata/shm major:0 minor:216 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/184ca9e8b8f3f0038d120e2dc91ba930384fad0d99b09baf7cdad8b2b34e0f82/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/184ca9e8b8f3f0038d120e2dc91ba930384fad0d99b09baf7cdad8b2b34e0f82/userdata/shm major:0 minor:88 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/28fcb0717813a29dc1866541bc3dbb0f0258922adc7924ce1ef6a82d0c101225/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/28fcb0717813a29dc1866541bc3dbb0f0258922adc7924ce1ef6a82d0c101225/userdata/shm major:0 minor:47 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/34704644b2f8aaf65517def9739a5a94a2ef4102e5f63eb6c50786d5d9f25147/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/34704644b2f8aaf65517def9739a5a94a2ef4102e5f63eb6c50786d5d9f25147/userdata/shm major:0 minor:68 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/36282a0a35d734aa5ae4edab1c2cfe275e0f3f20e9395e246cb64b674cc64752/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/36282a0a35d734aa5ae4edab1c2cfe275e0f3f20e9395e246cb64b674cc64752/userdata/shm major:0 minor:76 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/39292c76981dbdc83a09b7bd744ad4ab390d9da06ab541901ca5de7d61391dcb/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/39292c76981dbdc83a09b7bd744ad4ab390d9da06ab541901ca5de7d61391dcb/userdata/shm major:0 minor:92 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/463c6701469021f7bcc9b98670fa5c431575005397ec36ed2c36164ecc754824/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/463c6701469021f7bcc9b98670fa5c431575005397ec36ed2c36164ecc754824/userdata/shm major:0 minor:202 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/5539d378b1542324519d761500f1f41db809528c714c434124937ae73d77ecb8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/5539d378b1542324519d761500f1f41db809528c714c434124937ae73d77ecb8/userdata/shm major:0 minor:72 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/718ae0f8599bafd81a81c85617084005d43c05ef02acf0fca6530b8df0ae5a0f/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/718ae0f8599bafd81a81c85617084005d43c05ef02acf0fca6530b8df0ae5a0f/userdata/shm major:0 minor:96 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/82f83e072dbbfec7616c9c6533f29fc26d7d3ced2d9c309ff133c23d90e397f8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/82f83e072dbbfec7616c9c6533f29fc26d7d3ced2d9c309ff133c23d90e397f8/userdata/shm major:0 minor:78 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/c12ae599e6c930890621622a0cb873d074ecc075f38e97c1aba91f32c510fdd8/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/c12ae599e6c930890621622a0cb873d074ecc075f38e97c1aba91f32c510fdd8/userdata/shm major:0 minor:229 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/ee9534b936fc2c51176c28d0ce4ea47c7e9ff43500af65957a17de521caccfe0/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/ee9534b936fc2c51176c28d0ce4ea47c7e9ff43500af65957a17de521caccfe0/userdata/shm major:0 minor:48 fsType:tmpfs blockSize:0} /run/containers/storage/overlay-containers/f4fb571e9cb3fe2a0f336911a73b85beae343bd4305c83c039786b15981b271e/userdata/shm:{mountpoint:/run/containers/storage/overlay-containers/f4fb571e9cb3fe2a0f336911a73b85beae343bd4305c83c039786b15981b271e/userdata/shm major:0 minor:213 fsType:tmpfs blockSize:0} /tmp:{mountpoint:/tmp major:0 minor:32 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/00b87b36-9ce2-480a-9924-22b33256fe1e/volumes/kubernetes.io~projected/kube-api-access-fnfcz:{mountpoint:/var/lib/kubelet/pods/00b87b36-9ce2-480a-9924-22b33256fe1e/volumes/kubernetes.io~projected/kube-api-access-fnfcz major:0 minor:61 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/367d8d85-a5e8-462d-b1d7-91a01e62b726/volumes/kubernetes.io~secret/agent-certs:{mountpoint:/var/lib/kubelet/pods/367d8d85-a5e8-462d-b1d7-91a01e62b726/volumes/kubernetes.io~secret/agent-certs major:0 minor:55 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/6767bd29-3165-4e67-81b3-e320e8324034/volumes/kubernetes.io~projected/kube-api-access-wsvfn:{mountpoint:/var/lib/kubelet/pods/6767bd29-3165-4e67-81b3-e320e8324034/volumes/kubernetes.io~projected/kube-api-access-wsvfn major:0 minor:65 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/758467e6-69e4-4c1f-b344-e1e74497c298/volumes/kubernetes.io~projected/kube-api-access-4g4b6:{mountpoint:/var/lib/kubelet/pods/758467e6-69e4-4c1f-b344-e1e74497c298/volumes/kubernetes.io~projected/kube-api-access-4g4b6 major:0 minor:66 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8064da0c-fc5d-493c-bcdf-4327716d4cc2/volumes/kubernetes.io~projected/kube-api-access-kjmhh:{mountpoint:/var/lib/kubelet/pods/8064da0c-fc5d-493c-bcdf-4327716d4cc2/volumes/kubernetes.io~projected/kube-api-access-kjmhh major:0 minor:228 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/83455bdf-65dc-4f66-b943-55835e92b360/volumes/kubernetes.io~projected/kube-api-access-l4kfc:{mountpoint:/var/lib/kubelet/pods/83455bdf-65dc-4f66-b943-55835e92b360/volumes/kubernetes.io~projected/kube-api-access-l4kfc major:0 minor:220 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d/volumes/kubernetes.io~secret/original-pull-secret:{mountpoint:/var/lib/kubelet/pods/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d/volumes/kubernetes.io~secret/original-pull-secret major:0 minor:208 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/a974c7e3-bf97-4575-bec1-c09fcd018f0d/volumes/kubernetes.io~projected/kube-api-access-d7jnh:{mountpoint:/var/lib/kubelet/pods/a974c7e3-bf97-4575-bec1-c09fcd018f0d/volumes/kubernetes.io~projected/kube-api-access-d7jnh major:0 minor:215 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba5cbec9-7649-462a-b110-11ebcc5d6d78/volumes/kubernetes.io~projected/kube-api-access-kfzh6:{mountpoint:/var/lib/kubelet/pods/ba5cbec9-7649-462a-b110-11ebcc5d6d78/volumes/kubernetes.io~projected/kube-api-access-kfzh6 major:0 minor:201 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ba5cbec9-7649-462a-b110-11ebcc5d6d78/volumes/kubernetes.io~secret/cert:{mountpoint:/var/lib/kubelet/pods/ba5cbec9-7649-462a-b110-11ebcc5d6d78/volumes/kubernetes.io~secret/cert major:0 minor:45 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c2d7ba52-9a2b-4871-aca0-3c4e1934c572/volumes/kubernetes.io~projected/kube-api-access-5xrlg:{mountpoint:/var/lib/kubelet/pods/c2d7ba52-9a2b-4871-aca0-3c4e1934c572/volumes/kubernetes.io~projected/kube-api-access-5xrlg major:0 minor:67 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/volume-subpaths/run-systemd/ovnkube-controller/6:{mountpoint:/var/lib/kubelet/pods/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/volume-subpaths/run-systemd/ovnkube-controller/6 major:0 minor:26 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/volumes/kubernetes.io~projected/kube-api-access-tvrzx:{mountpoint:/var/lib/kubelet/pods/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/volumes/kubernetes.io~projected/kube-api-access-tvrzx major:0 minor:60 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/volumes/kubernetes.io~secret/ovn-node-metrics-cert:{mountpoint:/var/lib/kubelet/pods/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/volumes/kubernetes.io~secret/ovn-node-metrics-cert major:0 minor:58 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/ce8b7505-62a5-42f1-b432-d77200c21d2d/volumes/kubernetes.io~projected/kube-api-access-7zjwg:{mountpoint:/var/lib/kubelet/pods/ce8b7505-62a5-42f1-b432-d77200c21d2d/volumes/kubernetes.io~projected/kube-api-access-7zjwg major:0 minor:63 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/da9f1b66-0e1b-4c7e-a947-a897c00ccc19/volumes/kubernetes.io~projected/kube-api-access-kd2rw:{mountpoint:/var/lib/kubelet/pods/da9f1b66-0e1b-4c7e-a947-a897c00ccc19/volumes/kubernetes.io~projected/kube-api-access-kd2rw major:0 minor:221 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/dd733778-7002-4a26-a4e8-4525d8f6bee9/volumes/kubernetes.io~projected/kube-api-access-42fn2:{mountpoint:/var/lib/kubelet/pods/dd733778-7002-4a26-a4e8-4525d8f6bee9/volumes/kubernetes.io~projected/kube-api-access-42fn2 major:0 minor:64 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f26fab1e-e6d4-4caa-bef6-81a09d78c0de/volumes/kubernetes.io~projected/kube-api-access-skhvr:{mountpoint:/var/lib/kubelet/pods/f26fab1e-e6d4-4caa-bef6-81a09d78c0de/volumes/kubernetes.io~projected/kube-api-access-skhvr major:0 minor:59 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4b72e7f-cc4c-4662-a923-c184069aa118/volumes/kubernetes.io~empty-dir/etc-tuned:{mountpoint:/var/lib/kubelet/pods/f4b72e7f-cc4c-4662-a923-c184069aa118/volumes/kubernetes.io~empty-dir/etc-tuned major:0 minor:57 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4b72e7f-cc4c-4662-a923-c184069aa118/volumes/kubernetes.io~empty-dir/tmp:{mountpoint:/var/lib/kubelet/pods/f4b72e7f-cc4c-4662-a923-c184069aa118/volumes/kubernetes.io~empty-dir/tmp major:0 minor:56 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f4b72e7f-cc4c-4662-a923-c184069aa118/volumes/kubernetes.io~projected/kube-api-access-f59q8:{mountpoint:/var/lib/kubelet/pods/f4b72e7f-cc4c-4662-a923-c184069aa118/volumes/kubernetes.io~projected/kube-api-access-f59q8 major:0 minor:62 fsType:tmpfs blockSize:0} /var/lib/kubelet/pods/f59eba5d-a500-462e-9ecd-4ef9e553f873/volumes/kubernetes.io~projected/kube-api-access-cbbbk:{mountpoint:/var/lib/kubelet/pods/f59eba5d-a500-462e-9ecd-4ef9e553f873/volumes/kubernetes.io~projected/kube-api-access-cbbbk major:0 minor:197 fsType:tmpfs blockSize:0} composefs_0-34:{mountpoint:/ major:0 minor:34 fsType:overlay blockSize:0} overlay_0-100:{mountpoint:/var/lib/containers/storage/overlay/e64870204381c7a7f2555e3b5fe92951842ced33934d9f090c927aa4bb74212e/merged major:0 minor:100 fsType:overlay blockSize:0} overlay_0-102:{mountpoint:/var/lib/containers/storage/overlay/38f6c3a3bf1761682b345c03cb02c31f484b894296aec5d499ce5ae2e5601af5/merged major:0 minor:102 fsType:overlay blockSize:0} overlay_0-104:{mountpoint:/var/lib/containers/storage/overlay/0a6251ac8edb01e4a1a495ea3104a4be3776a4bde7a9167c9216fe870b0b5a1d/merged major:0 minor:104 fsType:overlay blockSize:0} overlay_0-106:{mountpoint:/var/lib/containers/storage/overlay/48a8a99b5aa38b5a1f8af93f2cf91f2afe257cea7b9b3fc9ad86c404aea4cfa8/merged major:0 minor:106 fsType:overlay blockSize:0} overlay_0-114:{mountpoint:/var/lib/containers/storage/overlay/cb4bed28be09871a074b3642832880027d77099e69a0ae8241aeb94dab27fdaf/merged major:0 minor:114 fsType:overlay blockSize:0} overlay_0-119:{mountpoint:/var/lib/containers/storage/overlay/48d5c82df7e650a089ea0fa91ed0bd4c1b6a9b1f380d8d8db83882fc5cc1bf60/merged major:0 minor:119 fsType:overlay blockSize:0} overlay_0-124:{mountpoint:/var/lib/containers/storage/overlay/69e9e5b9009b44f81cb13f7e89161acc514055b4129ec8240b07d4f06c225a45/merged major:0 minor:124 fsType:overlay blockSize:0} overlay_0-127:{mountpoint:/var/lib/containers/storage/overlay/065839dc8c2c6a550d10b437944e397361cfc89bf9ccb63c18a6d705e7669644/merged major:0 minor:127 fsType:overlay blockSize:0} overlay_0-132:{mountpoint:/var/lib/containers/storage/overlay/861acfcf2321b99e26e85c3154bd9202cc22f4d667e5041f027bd7d24464ecb9/merged major:0 minor:132 fsType:overlay blockSize:0} overlay_0-137:{mountpoint:/var/lib/containers/storage/overlay/48b947c743d65c0ae34868fbb47aeb890dde93159790d378edd81be829b75cfb/merged major:0 minor:137 fsType:overlay blockSize:0} overlay_0-140:{mountpoint:/var/lib/containers/storage/overlay/92e435ce4ca5afe43253221b15bf185141e8b9ccff9215e533a29abb68c8d01e/merged major:0 minor:140 fsType:overlay blockSize:0} overlay_0-144:{mountpoint:/var/lib/containers/storage/overlay/2850c96a313d433f759e56135d890d3c3b9d4452f3e507896aa256d912e71270/merged major:0 minor:144 fsType:overlay blockSize:0} overlay_0-152:{mountpoint:/var/lib/containers/storage/overlay/d5d50d36d455b3124bba1a4682c6b556a344e4c0f72561f6ab14472bbe327aa9/merged major:0 minor:152 fsType:overlay blockSize:0} overlay_0-157:{mountpoint:/var/lib/containers/storage/overlay/df02c5a738481c502ee0fbb85ca8fb2c47529fde2b1bfbb73efe51cb364c310b/merged major:0 minor:157 fsType:overlay blockSize:0} overlay_0-162:{mountpoint:/var/lib/containers/storage/overlay/e34d2a2f5077d6161aee9054ac97bc88dad0afe4dadc0770608b95d5aacf2f11/merged major:0 minor:162 fsType:overlay blockSize:0} overlay_0-167:{mountpoint:/var/lib/containers/storage/overlay/1b551b04f397d2a17bb4b72d2a9e13ea7b255a0c2887bd5f17c4e17a3c5d3890/merged major:0 minor:167 fsType:overlay blockSize:0} overlay_0-172:{mountpoint:/var/lib/containers/storage/overlay/87c11e77a69c1256c563023edeadf4b64a91805f1340898de6ce44397e54a787/merged major:0 minor:172 fsType:overlay blockSize:0} overlay_0-177:{mountpoint:/var/lib/containers/storage/overlay/6938a4b799445ead8836745535273347f3c908c027c27cae3317fe23a343f2cc/merged major:0 minor:177 fsType:overlay blockSize:0} overlay_0-182:{mountpoint:/var/lib/containers/storage/overlay/349bb891311b428d7245edc3a45686b0ebfaa1beb5ddb02e1c5186f29d1a0a38/merged major:0 minor:182 fsType:overlay blockSize:0} overlay_0-187:{mountpoint:/var/lib/containers/storage/overlay/1550c3e2e2a45dcfd84afdd33aa3690aa5dcb9b9475cb07cc27b0e0b34c41bb3/merged major:0 minor:187 fsType:overlay blockSize:0} overlay_0-192:{mountpoint:/var/lib/containers/storage/overlay/37c279f7138b64cf848e046dd0159412d2f60a6e3f561819a069d84c7835dd5e/merged major:0 minor:192 fsType:overlay blockSize:0} overlay_0-204:{mountpoint:/var/lib/containers/storage/overlay/b08034c65637061c7d06856c8763d8af38fa0083db3da8a46ee71187bc8aa3f5/merged major:0 minor:204 fsType:overlay blockSize:0} overlay_0-206:{mountpoint:/var/lib/containers/storage/overlay/ec5326cd4c57e5289c85be1ed540648b29113e91e4b3f4a2026c4c66a8e87d65/merged major:0 minor:206 fsType:overlay blockSize:0} overlay_0-218:{mountpoint:/var/lib/containers/storage/overlay/88bc22348df8e280411872f066dbf14feb3968b22f192d53027960ffae02080e/merged major:0 minor:218 fsType:overlay blockSize:0} overlay_0-224:{mountpoint:/var/lib/containers/storage/overlay/08a91d46928182b4e183ee7ace17ce789177b554acc520da9618f6d1c993924e/merged major:0 minor:224 fsType:overlay blockSize:0} overlay_0-226:{mountpoint:/var/lib/containers/storage/overlay/038a6c3c692ce53f7d2d34f09f7755b85ebdf1abcb04b16d6d77b0128a033ab2/merged major:0 minor:226 fsType:overlay blockSize:0} overlay_0-231:{mountpoint:/var/lib/containers/storage/overlay/aff244db02eacd0334ea3e804c3284703be41cda08f02e8f8d0a2997784e5b04/merged major:0 minor:231 fsType:overlay blockSize:0} overlay_0-233:{mountpoint:/var/lib/containers/storage/overlay/932ab5e5702f5c7c79c2d32910577875654df1477eac0759bb03b85f11ce0d04/merged major:0 minor:233 fsType:overlay blockSize:0} overlay_0-235:{mountpoint:/var/lib/containers/storage/overlay/72b9d04664a9c27fd69db6918064763b2008cae61f0f8b2e1653be8139dae121/merged major:0 minor:235 fsType:overlay blockSize:0} overlay_0-241:{mountpoint:/var/lib/containers/storage/overlay/68adcccdea3eba205d654c160d713ccd9fe3f136335e3c8c4fd9800b1d4da9c1/merged major:0 minor:241 fsType:overlay blockSize:0} overlay_0-247:{mountpoint:/var/lib/containers/storage/overlay/b7702443b08031bf70dfb72c9da1f832917de0a24b9b53011dbe509513501266/merged major:0 minor:247 fsType:overlay blockSize:0} overlay_0-51:{mountpoint:/var/lib/containers/storage/overlay/9dcf2fbc8f2a12b867ec625655279863edbfabf7bb03acffd8482f4d3a0eae26/merged major:0 minor:51 fsType:overlay blockSize:0} overlay_0-53:{mountpoint:/var/lib/containers/storage/overlay/ec89134dec345974daece69f9b8dd6137e843685bffddf149e15d295f298817e/merged major:0 minor:53 fsType:overlay blockSize:0} overlay_0-70:{mountpoint:/var/lib/containers/storage/overlay/1b20b5f4508512ef1b3778f562c5f97da26da6bf4d0856c4ef9a07f44819b3f0/merged major:0 minor:70 fsType:overlay blockSize:0} overlay_0-74:{mountpoint:/var/lib/containers/storage/overlay/a1ee19348961dae79c087e8ac3bf9e59ed7a33f37091eafee1f692eee68bc0a0/merged major:0 minor:74 fsType:overlay blockSize:0} overlay_0-79:{mountpoint:/var/lib/containers/storage/overlay/bd48af5b0a05e1435415768044430abaf131384d551707641cd919d54da040a4/merged major:0 minor:79 fsType:overlay blockSize:0} overlay_0-82:{mountpoint:/var/lib/containers/storage/overlay/a7f26589ec75d499102c1e32470fe7d9c2e016680764619bc4f468b50424ee5f/merged major:0 minor:82 fsType:overlay blockSize:0} overlay_0-86:{mountpoint:/var/lib/containers/storage/overlay/6db68c524c0be1eea698df87a8b9a3de739d2a7cafcb2c4289aec09a3b1cfe4f/merged major:0 minor:86 fsType:overlay blockSize:0} overlay_0-90:{mountpoint:/var/lib/containers/storage/overlay/4e361aab9892c59ae1c857a4c3f26675ab711d42dbb2eea22c0038a24983687a/merged major:0 minor:90 fsType:overlay blockSize:0} overlay_0-94:{mountpoint:/var/lib/containers/storage/overlay/9f3bb7fa13dcbf6c99ec343710a35437e7f61d66d1bdac63538a05c396354291/merged major:0 minor:94 fsType:overlay blockSize:0} overlay_0-98:{mountpoint:/var/lib/containers/storage/overlay/e63da92cc01fe6e396c33175c254611578c02ed94dc2baa9164216cbb22bf034/merged major:0 minor:98 fsType:overlay blockSize:0}]
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.564619    4456 manager.go:217] Machine: {Timestamp:2026-11-02 17:29:43.564176559 +0000 UTC m=+0.055239711 CPUVendorID:GenuineIntel NumCores:8 NumPhysicalCores:4 NumSockets:1 CpuFrequency:3100023 MemoryCapacity:32812019712 SwapCapacity:0 MemoryByType:map[] NVMInfo:{MemoryModeCapacity:0 AppDirectModeCapacity:0 AvgPowerBudget:0} HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] MachineID:ec2eefb5bbdc0967233fbc8f25552fee SystemUUID:ec2eefb5-bbdc-0967-233f-bc8f25552fee BootID:c30eb82f-0141-4319-9151-e90a0e6750c5 Filesystems:[{Device:/run/containers/storage/overlay-containers/ee9534b936fc2c51176c28d0ce4ea47c7e9ff43500af65957a17de521caccfe0/userdata/shm DeviceMajor:0 DeviceMinor:48 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-51 DeviceMajor:0 DeviceMinor:51 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-86 DeviceMajor:0 DeviceMinor:86 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-157 DeviceMajor:0 DeviceMinor:157 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-162 DeviceMajor:0 DeviceMinor:162 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba5cbec9-7649-462a-b110-11ebcc5d6d78/volumes/kubernetes.io~secret/cert DeviceMajor:0 DeviceMinor:45 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/6767bd29-3165-4e67-81b3-e320e8324034/volumes/kubernetes.io~projected/kube-api-access-wsvfn DeviceMajor:0 DeviceMinor:65 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/0e45823f1f3e78e26f6c3e2889edb7592f7debd8b8dbcc4dcf99f8f291f90da1/userdata/shm DeviceMajor:0 DeviceMinor:84 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/718ae0f8599bafd81a81c85617084005d43c05ef02acf0fca6530b8df0ae5a0f/userdata/shm DeviceMajor:0 DeviceMinor:96 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-98 DeviceMajor:0 DeviceMinor:98 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-132 DeviceMajor:0 DeviceMinor:132 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/f4fb571e9cb3fe2a0f336911a73b85beae343bd4305c83c039786b15981b271e/userdata/shm DeviceMajor:0 DeviceMinor:213 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-235 DeviceMajor:0 DeviceMinor:235 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/36282a0a35d734aa5ae4edab1c2cfe275e0f3f20e9395e246cb64b674cc64752/userdata/shm DeviceMajor:0 DeviceMinor:76 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/367d8d85-a5e8-462d-b1d7-91a01e62b726/volumes/kubernetes.io~secret/agent-certs DeviceMajor:0 DeviceMinor:55 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4b72e7f-cc4c-4662-a923-c184069aa118/volumes/kubernetes.io~empty-dir/etc-tuned DeviceMajor:0 DeviceMinor:57 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/00b87b36-9ce2-480a-9924-22b33256fe1e/volumes/kubernetes.io~projected/kube-api-access-fnfcz DeviceMajor:0 DeviceMinor:61 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-82 DeviceMajor:0 DeviceMinor:82 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ce8b7505-62a5-42f1-b432-d77200c21d2d/volumes/kubernetes.io~projected/kube-api-access-7zjwg DeviceMajor:0 DeviceMinor:63 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-119 DeviceMajor:0 DeviceMinor:119 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-127 DeviceMajor:0 DeviceMinor:127 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-167 DeviceMajor:0 DeviceMinor:167 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-192 DeviceMajor:0 DeviceMinor:192 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-231 DeviceMajor:0 DeviceMinor:231 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-233 DeviceMajor:0 DeviceMinor:233 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:/var/lib/kubelet/pods/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/volume-subpaths/run-systemd/ovnkube-controller/6 DeviceMajor:0 DeviceMinor:26 Capacity:6562406400 Type:vfs Inodes:819200 HasInodes:true} {Device:overlay_0-204 DeviceMajor:0 DeviceMinor:204 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/c2d7ba52-9a2b-4871-aca0-3c4e1934c572/volumes/kubernetes.io~projected/kube-api-access-5xrlg DeviceMajor:0 DeviceMinor:67 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-137 DeviceMajor:0 DeviceMinor:137 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/83455bdf-65dc-4f66-b943-55835e92b360/volumes/kubernetes.io~projected/kube-api-access-l4kfc DeviceMajor:0 DeviceMinor:220 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/00c2ae5d68441defef474549a48ed21d2249faeefb1b703c42569e621f3a57d2/userdata/shm DeviceMajor:0 DeviceMinor:222 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-226 DeviceMajor:0 DeviceMinor:226 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/nvme0n1p3 DeviceMajor:259 DeviceMinor:3 Capacity:366869504 Type:vfs Inodes:98304 HasInodes:true} {Device:/run/containers/storage/overlay-containers/82f83e072dbbfec7616c9c6533f29fc26d7d3ced2d9c309ff133c23d90e397f8/userdata/shm DeviceMajor:0 DeviceMinor:78 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-106 DeviceMajor:0 DeviceMinor:106 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/da9f1b66-0e1b-4c7e-a947-a897c00ccc19/volumes/kubernetes.io~projected/kube-api-access-kd2rw DeviceMajor:0 DeviceMinor:221 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/tmp DeviceMajor:0 DeviceMinor:32 Capacity:16406011904 Type:vfs Inodes:1048576 HasInodes:true} {Device:/run/containers/storage/overlay-containers/34704644b2f8aaf65517def9739a5a94a2ef4102e5f63eb6c50786d5d9f25147/userdata/shm DeviceMajor:0 DeviceMinor:68 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-70 DeviceMajor:0 DeviceMinor:70 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-152 DeviceMajor:0 DeviceMinor:152 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-177 DeviceMajor:0 DeviceMinor:177 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d/volumes/kubernetes.io~secret/original-pull-secret DeviceMajor:0 DeviceMinor:208 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/volumes/kubernetes.io~secret/ovn-node-metrics-cert DeviceMajor:0 DeviceMinor:58 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:composefs_0-34 DeviceMajor:0 DeviceMinor:34 Capacity:6422528 Type:vfs Inodes:18446744073709551615 HasInodes:true} {Device:overlay_0-102 DeviceMajor:0 DeviceMinor:102 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f26fab1e-e6d4-4caa-bef6-81a09d78c0de/volumes/kubernetes.io~projected/kube-api-access-skhvr DeviceMajor:0 DeviceMinor:59 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/volumes/kubernetes.io~projected/kube-api-access-tvrzx DeviceMajor:0 DeviceMinor:60 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/184ca9e8b8f3f0038d120e2dc91ba930384fad0d99b09baf7cdad8b2b34e0f82/userdata/shm DeviceMajor:0 DeviceMinor:88 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-90 DeviceMajor:0 DeviceMinor:90 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-94 DeviceMajor:0 DeviceMinor:94 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-144 DeviceMajor:0 DeviceMinor:144 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-182 DeviceMajor:0 DeviceMinor:182 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/15998cddba7811cb63969e4d4ccd9d484b4aa121c0a8c645977cfb696a73e5b5/userdata/shm DeviceMajor:0 DeviceMinor:216 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4b72e7f-cc4c-4662-a923-c184069aa118/volumes/kubernetes.io~empty-dir/tmp DeviceMajor:0 DeviceMinor:56 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/dd733778-7002-4a26-a4e8-4525d8f6bee9/volumes/kubernetes.io~projected/kube-api-access-42fn2 DeviceMajor:0 DeviceMinor:64 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/758467e6-69e4-4c1f-b344-e1e74497c298/volumes/kubernetes.io~projected/kube-api-access-4g4b6 DeviceMajor:0 DeviceMinor:66 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-79 DeviceMajor:0 DeviceMinor:79 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/39292c76981dbdc83a09b7bd744ad4ab390d9da06ab541901ca5de7d61391dcb/userdata/shm DeviceMajor:0 DeviceMinor:92 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-124 DeviceMajor:0 DeviceMinor:124 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/463c6701469021f7bcc9b98670fa5c431575005397ec36ed2c36164ecc754824/userdata/shm DeviceMajor:0 DeviceMinor:202 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-247 DeviceMajor:0 DeviceMinor:247 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-100 DeviceMajor:0 DeviceMinor:100 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-140 DeviceMajor:0 DeviceMinor:140 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-172 DeviceMajor:0 DeviceMinor:172 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/f59eba5d-a500-462e-9ecd-4ef9e553f873/volumes/kubernetes.io~projected/kube-api-access-cbbbk DeviceMajor:0 DeviceMinor:197 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-224 DeviceMajor:0 DeviceMinor:224 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/8064da0c-fc5d-493c-bcdf-4327716d4cc2/volumes/kubernetes.io~projected/kube-api-access-kjmhh DeviceMajor:0 DeviceMinor:228 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-53 DeviceMajor:0 DeviceMinor:53 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/c12ae599e6c930890621622a0cb873d074ecc075f38e97c1aba91f32c510fdd8/userdata/shm DeviceMajor:0 DeviceMinor:229 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-114 DeviceMajor:0 DeviceMinor:114 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-74 DeviceMajor:0 DeviceMinor:74 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/dev/shm DeviceMajor:0 DeviceMinor:24 Capacity:16406007808 Type:vfs Inodes:4005373 HasInodes:true} {Device:/dev/nvme0n1p4 DeviceMajor:259 DeviceMinor:4 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-187 DeviceMajor:0 DeviceMinor:187 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-206 DeviceMajor:0 DeviceMinor:206 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/a974c7e3-bf97-4575-bec1-c09fcd018f0d/volumes/kubernetes.io~projected/kube-api-access-d7jnh DeviceMajor:0 DeviceMinor:215 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-218 DeviceMajor:0 DeviceMinor:218 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:overlay_0-241 DeviceMajor:0 DeviceMinor:241 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/run/containers/storage/overlay-containers/28fcb0717813a29dc1866541bc3dbb0f0258922adc7924ce1ef6a82d0c101225/userdata/shm DeviceMajor:0 DeviceMinor:47 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:/var/lib/kubelet/pods/f4b72e7f-cc4c-4662-a923-c184069aa118/volumes/kubernetes.io~projected/kube-api-access-f59q8 DeviceMajor:0 DeviceMinor:62 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true} {Device:/run/containers/storage/overlay-containers/5539d378b1542324519d761500f1f41db809528c714c434124937ae73d77ecb8/userdata/shm DeviceMajor:0 DeviceMinor:72 Capacity:67108864 Type:vfs Inodes:4005373 HasInodes:true} {Device:overlay_0-104 DeviceMajor:0 DeviceMinor:104 Capacity:128243970048 Type:vfs Inodes:62651840 HasInodes:true} {Device:/var/lib/kubelet/pods/ba5cbec9-7649-462a-b110-11ebcc5d6d78/volumes/kubernetes.io~projected/kube-api-access-kfzh6 DeviceMajor:0 DeviceMinor:201 Capacity:31633420288 Type:vfs Inodes:4005373 HasInodes:true}] DiskMap:map[259:0:{Name:nvme0n1 Major:259 Minor:0 Size:128849018880 Scheduler:none}] NetworkDevices:[{Name:00c2ae5d68441de MacAddress:82:9e:33:25:1f:63 Speed:10000 Mtu:8901} {Name:15998cddba7811c MacAddress:c6:56:98:30:fe:18 Speed:10000 Mtu:8901} {Name:463c6701469021f MacAddress:f6:55:61:dd:ca:26 Speed:10000 Mtu:8901} {Name:br-ex MacAddress:02:ff:ce:93:b2:55 Speed:0 Mtu:9001} {Name:br-int MacAddress:22:e3:a1:c1:c0:65 Speed:0 Mtu:8901} {Name:c12ae599e6c9308 MacAddress:82:82:c3:c3:49:fa Speed:10000 Mtu:8901} {Name:ens5 MacAddress:02:ff:ce:93:b2:55 Speed:0 Mtu:9001} {Name:f4fb571e9cb3fe2 MacAddress:5a:bb:dc:5e:55:a6 Speed:10000 Mtu:8901} {Name:genev_sys_6081 MacAddress:ce:2c:63:d7:b9:ac Speed:0 Mtu:65000} {Name:ovn-k8s-mp0 MacAddress:0a:58:0a:86:00:02 Speed:0 Mtu:8901} {Name:ovs-system MacAddress:ce:07:77:91:5b:88 Speed:0 Mtu:1500}] Topology:[{Id:0 Memory:32812019712 HugePages:[{PageSize:1048576 NumPages:0} {PageSize:2048 NumPages:0}] Cores:[{Id:0 Threads:[0 4] Caches:[{Id:0 Size:32768 Type:Data Level:1} {Id:0 Size:32768 Type:Instruction Level:1} {Id:0 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:1 Threads:[1 5] Caches:[{Id:1 Size:32768 Type:Data Level:1} {Id:1 Size:32768 Type:Instruction Level:1} {Id:1 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:2 Threads:[2 6] Caches:[{Id:2 Size:32768 Type:Data Level:1} {Id:2 Size:32768 Type:Instruction Level:1} {Id:2 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:} {Id:3 Threads:[3 7] Caches:[{Id:3 Size:32768 Type:Data Level:1} {Id:3 Size:32768 Type:Instruction Level:1} {Id:3 Size:1048576 Type:Unified Level:2}] UncoreCaches:[] SocketID:0 BookID: DrawerID:}] Caches:[{Id:0 Size:34603008 Type:Unified Level:3}] Distances:[10]}] CloudProvider:Unknown InstanceType:Unknown InstanceID:None}
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.564929    4456 manager_no_libpfm.go:29] cAdvisor is build without cgo and/or libpfm support. Perf event counters are not available.
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.565017    4456 manager.go:233] Version: {KernelVersion:5.14.0-687.60.1.el9_8.x86_64 ContainerOsVersion:Red Hat Enterprise Linux CoreOS 9.8.20261023-0 (Plow) DockerVersion: DockerAPIVersion: CadvisorVersion: CadvisorRevision:}
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.565348    4456 container_manager_linux.go:275] "Container manager verified user specified cgroup-root exists" cgroupRoot=[]
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.565375    4456 container_manager_linux.go:280] "Creating Container Manager object based on Node Config" nodeConfig={"NodeName":"ip-10-0-141-87.ec2.internal","RuntimeCgroupsName":"/system.slice/crio.service","SystemCgroupsName":"/system.slice","KubeletCgroupsName":"","KubeletOOMScoreAdj":-999,"ContainerRuntime":"","CgroupsPerQOS":true,"CgroupRoot":"/","CgroupDriver":"systemd","KubeletRootDir":"/var/lib/kubelet","ProtectKernelDefaults":true,"KubeReservedCgroupName":"","SystemReservedCgroupName":"/system.slice","ReservedSystemCPUs":{},"EnforceNodeAllocatable":{"pods":{},"system-reserved-compressible":{}},"KubeReserved":null,"SystemReserved":{"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"},"HardEvictionThresholds":[{"Signal":"imagefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.15},"GracePeriod":0,"MinReclaim":null},{"Signal":"imagefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null},{"Signal":"memory.available","Operator":"LessThan","Value":{"Quantity":"100Mi","Percentage":0},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.available","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.1},"GracePeriod":0,"MinReclaim":null},{"Signal":"nodefs.inodesFree","Operator":"LessThan","Value":{"Quantity":null,"Percentage":0.05},"GracePeriod":0,"MinReclaim":null}],"QOSReserved":{},"CPUManagerPolicy":"none","CPUManagerPolicyOptions":null,"TopologyManagerScope":"container","CPUManagerReconcilePeriod":10000000000,"MemoryManagerPolicy":"None","MemoryManagerReservedMemory":null,"PodPidsLimit":4096,"EnforceCPULimits":true,"CPUCFSQuotaPeriod":100000000,"TopologyManagerPolicy":"none","TopologyManagerPolicyOptions":null,"CgroupVersion":2}
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.565495    4456 topology_manager.go:143] "Creating topology manager with none policy"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.565503    4456 container_manager_linux.go:311] "Creating device plugin manager"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.565513    4456 manager.go:144] "Creating Device Plugin manager" path="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.565529    4456 server.go:74] "Creating device plugin registration server" version="v1beta1" socket="/var/lib/kubelet/device-plugins/kubelet.sock"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.565590    4456 container_manager_linux.go:320] "Creating Dynamic Resource Allocation (DRA) manager"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.565802    4456 state_mem.go:41] "Initialized" logger="CPUManager state memory"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.565968    4456 server.go:1235] "Using root directory" path="/var/lib/kubelet"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.566003    4456 kubelet.go:493] "Attempting to sync node with API server"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.566013    4456 kubelet.go:394] "Adding static pod path" path="/etc/kubernetes/manifests"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.566028    4456 file.go:69] "Watching path" path="/etc/kubernetes/manifests"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.566038    4456 kubelet.go:405] "Adding apiserver pod source"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.566046    4456 apiserver.go:42] "Waiting for node sync before watching apiserver pods"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.566593    4456 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.566602    4456 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.567103    4456 kuberuntime_manager.go:304] "Container runtime initialized" containerRuntime="cri-o" version="1.35.10-3.rhaos4.22.git9383502.el9" apiVersion="v1"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.568893    4456 state_checkpoint.go:81] "State checkpoint: restored pod resource state from checkpoint"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.568902    4456 state_mem.go:40] "Initialized new in-memory state store for pod resource information tracking"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.568986    4456 certificate_store.go:147] "Loading cert/key pair from a file" filePath="/var/lib/kubelet/pki/kubelet-server-current.pem"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569155    4456 kubelet.go:963] "Not starting ClusterTrustBundle informer because we are in static kubelet mode or the ClusterTrustBundleProjection featuregate is disabled"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569182    4456 kubelet.go:990] "Not starting PodCertificateRequest manager because we are in static kubelet mode or the PodCertificateProjection feature gate is disabled"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569263    4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/portworx-volume"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569275    4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/empty-dir"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569281    4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/git-repo"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569287    4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/host-path"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569293    4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/nfs"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569300    4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/secret"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569313    4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/iscsi"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569318    4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/downward-api"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569325    4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/fc"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569331    4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/configmap"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569343    4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/projected"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569352    4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/local-volume"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569380    4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/csi"
Nov 02 17:29:43.569382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.569386    4456 plugins.go:610] "Loaded volume plugin" pluginName="kubernetes.io/image"
Nov 02 17:29:43.572561 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.572547    4456 server.go:1263] "Started kubelet"
Nov 02 17:29:43.572668 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.572617    4456 server.go:182] "Starting to listen" address="0.0.0.0" port=10250
Nov 02 17:29:43.572726 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.572649    4456 ratelimit.go:56] "Setting rate limiting for endpoint" service="podresources" qps=100 burstTokens=10
Nov 02 17:29:43.572776 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.572723    4456 server_v1.go:49] "podresources" method="list" useActivePods=true
Nov 02 17:29:43.573116 ip-10-0-141-87 systemd[1]: Started Kubernetes Kubelet.
Nov 02 17:29:43.573442 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.573118    4456 server.go:254] "Starting to serve the podresources API" endpoint="unix:/var/lib/kubelet/pod-resources/kubelet.sock"
Nov 02 17:29:43.578797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.576624    4456 server.go:317] "Adding debug handlers to kubelet server"
Nov 02 17:29:43.578797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.578085    4456 fs_resource_analyzer.go:69] "Starting FS ResourceAnalyzer"
Nov 02 17:29:43.578797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.578361    4456 certificate_manager.go:422] "Certificate rotation is enabled" logger="kubernetes.io/kubelet-serving"
Nov 02 17:29:43.578797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.578598    4456 volume_manager.go:309] "The desired_state_of_world populator starts"
Nov 02 17:29:43.578797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.578582    4456 certificate_manager.go:715] "Certificate rotation deadline determined" logger="kubernetes.io/kubelet-serving" expiration="2028-11-01 17:24:05 +0000 UTC" deadline="2028-08-20 15:54:02.771728544 +0000 UTC"
Nov 02 17:29:43.578797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.578615    4456 certificate_manager.go:431] "Waiting for next certificate rotation" logger="kubernetes.io/kubelet-serving" sleep="15766h24m19.19311638s"
Nov 02 17:29:43.578797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.578619    4456 desired_state_of_world_populator.go:146] "Desired state populator starts to run"
Nov 02 17:29:43.578797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.578608    4456 volume_manager.go:311] "Starting Kubelet Volume Manager"
Nov 02 17:29:43.578797 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:43.578653    4456 kubelet_node_status.go:442] "Error getting the current node from lister" err="node \"ip-10-0-141-87.ec2.internal\" not found"
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581125    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f59eba5d-a500-462e-9ecd-4ef9e553f873" volumeName="kubernetes.io/projected/f59eba5d-a500-462e-9ecd-4ef9e553f873-kube-api-access-cbbbk" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581165    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="a974c7e3-bf97-4575-bec1-c09fcd018f0d" volumeName="kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581181    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c86c4f40-2978-40f0-bfc1-c3cfffca0d8c" volumeName="kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovnkube-script-lib" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581193    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="dd733778-7002-4a26-a4e8-4525d8f6bee9" volumeName="kubernetes.io/projected/dd733778-7002-4a26-a4e8-4525d8f6bee9-kube-api-access-42fn2" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581206    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="758467e6-69e4-4c1f-b344-e1e74497c298" volumeName="kubernetes.io/projected/758467e6-69e4-4c1f-b344-e1e74497c298-kube-api-access-4g4b6" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581214    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8064da0c-fc5d-493c-bcdf-4327716d4cc2" volumeName="kubernetes.io/projected/8064da0c-fc5d-493c-bcdf-4327716d4cc2-kube-api-access-kjmhh" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581222    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83455bdf-65dc-4f66-b943-55835e92b360" volumeName="kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-trusted-ca-bundle" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581229    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" volumeName="kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-whereabouts-flatfile-configmap" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581143    4456 factory.go:55] Registering systemd factory
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581244    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c86c4f40-2978-40f0-bfc1-c3cfffca0d8c" volumeName="kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-env-overrides" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581258    4456 factory.go:223] Registration of the systemd container factory successfully
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581257    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6767bd29-3165-4e67-81b3-e320e8324034" volumeName="kubernetes.io/projected/6767bd29-3165-4e67-81b3-e320e8324034-kube-api-access-wsvfn" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581316    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce8b7505-62a5-42f1-b432-d77200c21d2d" volumeName="kubernetes.io/empty-dir/ce8b7505-62a5-42f1-b432-d77200c21d2d-tmp-dir" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581341    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f26fab1e-e6d4-4caa-bef6-81a09d78c0de" volumeName="kubernetes.io/projected/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-kube-api-access-skhvr" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581360    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d" volumeName="kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581374    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00b87b36-9ce2-480a-9924-22b33256fe1e" volumeName="kubernetes.io/projected/00b87b36-9ce2-480a-9924-22b33256fe1e-kube-api-access-fnfcz" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581389    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="367d8d85-a5e8-462d-b1d7-91a01e62b726" volumeName="kubernetes.io/secret/367d8d85-a5e8-462d-b1d7-91a01e62b726-agent-certs" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581400    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="6767bd29-3165-4e67-81b3-e320e8324034" volumeName="kubernetes.io/configmap/6767bd29-3165-4e67-81b3-e320e8324034-iptables-alerter-script" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581414    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba5cbec9-7649-462a-b110-11ebcc5d6d78" volumeName="kubernetes.io/projected/ba5cbec9-7649-462a-b110-11ebcc5d6d78-kube-api-access-kfzh6" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581439    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" volumeName="kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cni-sysctl-allowlist" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581452    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c86c4f40-2978-40f0-bfc1-c3cfffca0d8c" volumeName="kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovnkube-config" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581463    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f26fab1e-e6d4-4caa-bef6-81a09d78c0de" volumeName="kubernetes.io/configmap/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-cni-binary-copy" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581475    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83455bdf-65dc-4f66-b943-55835e92b360" volumeName="kubernetes.io/projected/83455bdf-65dc-4f66-b943-55835e92b360-kube-api-access-l4kfc" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581487    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c86c4f40-2978-40f0-bfc1-c3cfffca0d8c" volumeName="kubernetes.io/secret/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovn-node-metrics-cert" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581500    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ce8b7505-62a5-42f1-b432-d77200c21d2d" volumeName="kubernetes.io/projected/ce8b7505-62a5-42f1-b432-d77200c21d2d-kube-api-access-7zjwg" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581511    4456 factory.go:153] Registering CRI-O factory
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581526    4456 factory.go:223] Registration of the crio container factory successfully
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581588    4456 factory.go:221] Registration of the containerd container factory failed: unable to create containerd client: containerd: cannot unix dial containerd api service: dial unix /run/containerd/containerd.sock: connect: no such file or directory
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581607    4456 factory.go:103] Registering Raw factory
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581623    4456 manager.go:1199] Started watching for new ooms in manager
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581949    4456 manager.go:319] Starting recovery of all containers
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.581513    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f26fab1e-e6d4-4caa-bef6-81a09d78c0de" volumeName="kubernetes.io/configmap/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-daemon-config" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583009    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4b72e7f-cc4c-4662-a923-c184069aa118" volumeName="kubernetes.io/empty-dir/f4b72e7f-cc4c-4662-a923-c184069aa118-tmp" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583034    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4b72e7f-cc4c-4662-a923-c184069aa118" volumeName="kubernetes.io/empty-dir/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-tuned" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583049    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f4b72e7f-cc4c-4662-a923-c184069aa118" volumeName="kubernetes.io/projected/f4b72e7f-cc4c-4662-a923-c184069aa118-kube-api-access-f59q8" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583064    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f59eba5d-a500-462e-9ecd-4ef9e553f873" volumeName="kubernetes.io/configmap/f59eba5d-a500-462e-9ecd-4ef9e553f873-config-volume" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583081    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" volumeName="kubernetes.io/projected/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-kube-api-access-5xrlg" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583092    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="f59eba5d-a500-462e-9ecd-4ef9e553f873" volumeName="kubernetes.io/empty-dir/f59eba5d-a500-462e-9ecd-4ef9e553f873-tmp-dir" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583105    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="367d8d85-a5e8-462d-b1d7-91a01e62b726" volumeName="kubernetes.io/configmap/367d8d85-a5e8-462d-b1d7-91a01e62b726-konnectivity-ca" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583116    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="ba5cbec9-7649-462a-b110-11ebcc5d6d78" volumeName="kubernetes.io/secret/ba5cbec9-7649-462a-b110-11ebcc5d6d78-cert" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583127    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da9f1b66-0e1b-4c7e-a947-a897c00ccc19" volumeName="kubernetes.io/empty-dir/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-data-volume" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583139    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83455bdf-65dc-4f66-b943-55835e92b360" volumeName="kubernetes.io/empty-dir/83455bdf-65dc-4f66-b943-55835e92b360-archives-path" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583150    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="00b87b36-9ce2-480a-9924-22b33256fe1e" volumeName="kubernetes.io/configmap/00b87b36-9ce2-480a-9924-22b33256fe1e-serviceca" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583162    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="83455bdf-65dc-4f66-b943-55835e92b360" volumeName="kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-service-ca-bundle" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583192    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" volumeName="kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cni-binary-copy" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583204    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="c86c4f40-2978-40f0-bfc1-c3cfffca0d8c" volumeName="kubernetes.io/projected/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-kube-api-access-tvrzx" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583217    4456 reconstruct.go:129] "Volume is marked as uncertain and added into the actual state" pod="" podName="da9f1b66-0e1b-4c7e-a947-a897c00ccc19" volumeName="kubernetes.io/projected/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-api-access-kd2rw" seLinuxMountContext=""
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583227    4456 reconstruct.go:97] "Volume reconstruction finished"
Nov 02 17:29:43.583436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.583234    4456 reconciler.go:29] "Reconciler: start to sync state"
Nov 02 17:29:43.592783 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.592626    4456 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv6"
Nov 02 17:29:43.606040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.606019    4456 kubelet_network_linux.go:54] "Initialized iptables rules." protocol="IPv4"
Nov 02 17:29:43.606040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.606040    4456 status_manager.go:249] "Starting to sync pod status with apiserver"
Nov 02 17:29:43.606138 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.606055    4456 kubelet.go:2521] "Starting kubelet main sync loop"
Nov 02 17:29:43.606138 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:43.606094    4456 kubelet.go:2545] "Skipping pod synchronization" err="[container runtime status check may not have completed yet, PLEG is not healthy: pleg has yet to be successful]"
Nov 02 17:29:43.622892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.622859    4456 generic.go:356] "Generic (PLEG): container finished" podID="375359043409c490a8ed0d21b8afeaf5" containerID="96cb6a32c416cf30649ef241147265a21068e42498a6ee126aaad556aba5179c" exitCode=0
Nov 02 17:29:43.630541 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.630522    4456 reconstruct.go:205] "DevicePaths of reconstructed volumes updated"
Nov 02 17:29:43.634677 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.634658    4456 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:29:43.634785 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.634717    4456 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:29:43.635603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.635579    4456 generic.go:356] "Generic (PLEG): container finished" podID="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" containerID="d5407571491be8eb6db5a9af854fe88a1ba9d3cab526078d243a673491607c1f" exitCode=0
Nov 02 17:29:43.635603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.635602    4456 generic.go:356] "Generic (PLEG): container finished" podID="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" containerID="859ebbb72520ce950d67654afed6c62c3bfe8c7861db1d0485a4ca0a53d9a58c" exitCode=0
Nov 02 17:29:43.635741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.635609    4456 generic.go:356] "Generic (PLEG): container finished" podID="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" containerID="3c53a4dfddf899285adb42dc751a3c0735ffe894c1506754ee9c0e85238f9d1f" exitCode=0
Nov 02 17:29:43.635741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.635615    4456 generic.go:356] "Generic (PLEG): container finished" podID="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" containerID="f375f7cea091ed446006811bbcf4e8902d0c6b9bf4b3e5f6d6ba9fd8cc34e740" exitCode=0
Nov 02 17:29:43.635741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.635622    4456 generic.go:356] "Generic (PLEG): container finished" podID="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" containerID="5ca757f8bc481eca00d6d4427507cd91996755dcb73a5bf277184dde2a3f42ef" exitCode=0
Nov 02 17:29:43.635741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.635627    4456 generic.go:356] "Generic (PLEG): container finished" podID="c2d7ba52-9a2b-4871-aca0-3c4e1934c572" containerID="1eb8c5ec34c78e0c0caeff100b0da211c968076bca614600a2d6282d3ab5c78c" exitCode=0
Nov 02 17:29:43.647393 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.647371    4456 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:29:43.647503 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.647436    4456 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 17:29:43.648222 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.648209    4456 manager.go:324] Recovery completed
Nov 02 17:29:43.662993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.662973    4456 cpu_manager.go:226] "Starting" policy="none"
Nov 02 17:29:43.662993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.662986    4456 cpu_manager.go:227] "Reconciling" reconcilePeriod="10s"
Nov 02 17:29:43.663109 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.663002    4456 state_mem.go:41] "Initialized" logger="CPUManager state checkpoint.CPUManager state memory"
Nov 02 17:29:43.663174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.663120    4456 state_mem.go:94] "Updated default CPUSet" logger="CPUManager state checkpoint.CPUManager state memory" cpuSet=""
Nov 02 17:29:43.663174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.663130    4456 state_mem.go:102] "Updated CPUSet assignments" logger="CPUManager state checkpoint.CPUManager state memory" assignments={}
Nov 02 17:29:43.663174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.663145    4456 state_checkpoint.go:142] "restored state from checkpoint" logger="CPUManager state checkpoint"
Nov 02 17:29:43.663174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.663150    4456 state_checkpoint.go:143] "defaultCPUSet" logger="CPUManager state checkpoint" defaultCpuSet=""
Nov 02 17:29:43.663174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.663155    4456 policy_none.go:50] "Start"
Nov 02 17:29:43.663174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.663163    4456 memory_manager.go:187] "Starting memorymanager" policy="None"
Nov 02 17:29:43.663174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.663171    4456 state_mem.go:36] "Initializing new in-memory state store" logger="Memory Manager state checkpoint"
Nov 02 17:29:43.663373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.663284    4456 state_mem.go:77] "Updated machine memory state" logger="Memory Manager state checkpoint"
Nov 02 17:29:43.663373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.663291    4456 state_checkpoint.go:85] "State checkpoint: restored state from checkpoint" logger="Memory Manager state checkpoint"
Nov 02 17:29:43.663373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.663296    4456 policy_none.go:44] "Start"
Nov 02 17:29:43.667280 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.667254    4456 node_container_manager_linux.go:130] "Enforcing system reserved compressible on cgroup" cgroupName="/system.slice" limits={"cpu":"500m","ephemeral-storage":"1Gi","memory":"1Gi"}
Nov 02 17:29:43.670525 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.670507    4456 manager.go:341] "Starting Device Plugin manager"
Nov 02 17:29:43.671197 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:43.670535    4456 manager.go:525] "Failed to read data from checkpoint" err="checkpoint is not found" checkpoint="kubelet_internal_checkpoint"
Nov 02 17:29:43.671197 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.671198    4456 server.go:87] "Starting device plugin registration server"
Nov 02 17:29:43.671516 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.671501    4456 eviction_manager.go:194] "Eviction manager: starting control loop"
Nov 02 17:29:43.671590 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.671515    4456 container_log_manager.go:146] "Initializing container log rotate workers" workers=1 monitorPeriod="10s"
Nov 02 17:29:43.671640 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.671608    4456 plugin_watcher.go:53] "Plugin Watcher Start" path="/var/lib/kubelet/plugins_registry"
Nov 02 17:29:43.671739 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.671708    4456 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 17:29:43.671791 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.671749    4456 plugin_manager.go:119] "The desired_state_of_world populator (plugin watcher) starts"
Nov 02 17:29:43.671791 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.671756    4456 plugin_manager.go:121] "Starting Kubelet Plugin Manager"
Nov 02 17:29:43.671951 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.671889    4456 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T17:29:43.671735938Z","UUID":"47a14e57-d0eb-4a1c-b8e1-d66369b29c2c","Handler":null,"Name":"","Endpoint":""}
Nov 02 17:29:43.673673 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:43.673655    4456 eviction_manager.go:272] "eviction manager: failed to check if we have separate container filesystem. Ignoring." err="non-existent label \"crio-containers\""
Nov 02 17:29:43.673795 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.673685    4456 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 17:29:43.673795 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.673706    4456 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 17:29:43.706971 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.706931    4456 kubelet.go:2608] "SyncLoop ADD" source="file" pods=["openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal","kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"]
Nov 02 17:29:43.707406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.707379    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" event={"ID":"375359043409c490a8ed0d21b8afeaf5","Type":"ContainerStarted","Data":"cc6296601e412b777d553d9bc2c1afdc92713fa62371b11edd8c04e5807abdd5"}
Nov 02 17:29:43.707467 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.707419    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" event={"ID":"375359043409c490a8ed0d21b8afeaf5","Type":"ContainerDied","Data":"96cb6a32c416cf30649ef241147265a21068e42498a6ee126aaad556aba5179c"}
Nov 02 17:29:43.707467 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.707441    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal" event={"ID":"375359043409c490a8ed0d21b8afeaf5","Type":"ContainerStarted","Data":"28fcb0717813a29dc1866541bc3dbb0f0258922adc7924ce1ef6a82d0c101225"}
Nov 02 17:29:43.707534 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.707474    4456 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.707534 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.707506    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal" event={"ID":"fbfcd267e5bdda32002f5b72a55d4954","Type":"ContainerStarted","Data":"032fcf72ec88e49360029b5271b400c196f1b8b14e5eb38e8730bd4f60cb83b2"}
Nov 02 17:29:43.707534 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.707526    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal" event={"ID":"fbfcd267e5bdda32002f5b72a55d4954","Type":"ContainerStarted","Data":"ee9534b936fc2c51176c28d0ce4ea47c7e9ff43500af65957a17de521caccfe0"}
Nov 02 17:29:43.707638 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.707631    4456 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.735727 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.735710    4456 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:29:43.735833 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:43.735750    4456 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.736889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.736877    4456 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:29:43.736928 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:43.736902    4456 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-87.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.772299 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.772270    4456 kubelet_node_status.go:388] "Setting node annotation to enable volume controller attach/detach"
Nov 02 17:29:43.773531 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.773506    4456 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 17:29:43.773628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.773541    4456 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 17:29:43.773628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.773555    4456 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID"
Nov 02 17:29:43.773628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.773594    4456 kubelet_node_status.go:77] "Attempting to register node" node="ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.816216 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.816156    4456 kubelet_node_status.go:126] "Node was previously registered" node="ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.816321 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.816222    4456 kubelet_node_status.go:80] "Successfully registered node" node="ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.830958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.830930    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/375359043409c490a8ed0d21b8afeaf5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"375359043409c490a8ed0d21b8afeaf5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.831046 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.830969    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/375359043409c490a8ed0d21b8afeaf5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"375359043409c490a8ed0d21b8afeaf5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.831046 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.831003    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbfcd267e5bdda32002f5b72a55d4954-config\") pod \"kube-apiserver-proxy-ip-10-0-141-87.ec2.internal\" (UID: \"fbfcd267e5bdda32002f5b72a55d4954\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.931762 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.931729    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/375359043409c490a8ed0d21b8afeaf5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"375359043409c490a8ed0d21b8afeaf5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.931762 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.931762    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/375359043409c490a8ed0d21b8afeaf5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"375359043409c490a8ed0d21b8afeaf5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.931963 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.931777    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbfcd267e5bdda32002f5b72a55d4954-config\") pod \"kube-apiserver-proxy-ip-10-0-141-87.ec2.internal\" (UID: \"fbfcd267e5bdda32002f5b72a55d4954\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.931963 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.931836    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/375359043409c490a8ed0d21b8afeaf5-var-lib-kubelet\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"375359043409c490a8ed0d21b8afeaf5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.931963 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.931860    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kube\" (UniqueName: \"kubernetes.io/host-path/375359043409c490a8ed0d21b8afeaf5-etc-kube\") pod \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" (UID: \"375359043409c490a8ed0d21b8afeaf5\") " pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:43.931963 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:43.931896    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/host-path/fbfcd267e5bdda32002f5b72a55d4954-config\") pod \"kube-apiserver-proxy-ip-10-0-141-87.ec2.internal\" (UID: \"fbfcd267e5bdda32002f5b72a55d4954\") " pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:44.567188 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.567153    4456 apiserver.go:52] "Watching apiserver"
Nov 02 17:29:44.620658 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.620627    4456 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 17:29:44.621130 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.621109    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2","openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n","kube-system/global-pull-secret-syncer-mmhd7","kube-system/konnectivity-agent-mxsjq","kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal","openshift-dns/dns-default-zmrdt","openshift-ingress-canary/ingress-canary-n4r8t","openshift-multus/multus-additional-cni-plugins-ngkbz","openshift-network-diagnostics/network-check-target-nttxj","openshift-insights/periodic-gathering-wphg6-wf57n","openshift-multus/multus-7qzkz","openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal","openshift-network-operator/iptables-alerter-dlp6w","openshift-cluster-node-tuning-operator/tuned-n9fk9","openshift-dns/node-resolver-hhsrb","openshift-ovn-kubernetes/ovnkube-node-44l94","openshift-insights/insights-runtime-extractor-6bfcf","openshift-multus/network-metrics-daemon-tk8ht","openshift-image-registry/node-ca-lhwgm"]
Nov 02 17:29:44.624676 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.624656    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:44.627099 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.627074    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:44.628784 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.628759    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:44.645194 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.645177    4456 kubelet.go:3368] "Creating a mirror pod for static pod" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:44.645317 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.645302    4456 kubelet.go:3368] "Creating a mirror pod for static pod" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:44.651358 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.651339    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 17:29:44.651658 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.651639    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ppwdb\""
Nov 02 17:29:44.651711 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.651681    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qbqb7\""
Nov 02 17:29:44.651711 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.651681    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4b9fl\""
Nov 02 17:29:44.651711 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.651701    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9lngf\""
Nov 02 17:29:44.651849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.651709    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jl8vw\""
Nov 02 17:29:44.651849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.651715    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bl7zd\""
Nov 02 17:29:44.663453 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.663437    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 17:29:44.663603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.663587    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 17:29:44.664048 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.664016    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 17:29:44.664048 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.664042    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 17:29:44.664183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.664123    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t2hrp\""
Nov 02 17:29:44.664183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.664129    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xdl26\""
Nov 02 17:29:44.664183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.664158    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn2p2\""
Nov 02 17:29:44.664183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.664160    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 17:29:44.664183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.664170    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"operator-dockercfg-c5qgh\""
Nov 02 17:29:44.664183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.664180    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zt2z5\""
Nov 02 17:29:44.664371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.664175    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mwsx7\""
Nov 02 17:29:44.664371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.664175    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-metrics-serving-cert\""
Nov 02 17:29:44.664371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.664241    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fw7vt\""
Nov 02 17:29:44.664371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.664313    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5nstj\""
Nov 02 17:29:44.666022 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.666009    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 17:29:44.666109 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.666097    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rj9zp\""
Nov 02 17:29:44.680277 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.680261    4456 desired_state_of_world_populator.go:154] "Finished populating initial desired state of world"
Nov 02 17:29:44.690401 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.690382    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vjnnw\""
Nov 02 17:29:44.690401 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.690394    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vqf9q\""
Nov 02 17:29:44.702934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.702913    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 17:29:44.703078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.703063    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 17:29:44.703153 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.703103    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 17:29:44.703153 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.703143    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 17:29:44.703260 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.703142    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 17:29:44.703260 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.703163    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 17:29:44.703260 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.703147    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 17:29:44.703260 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.703180    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:29:44.703260 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.703186    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 17:29:44.703260 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.703200    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 17:29:44.703260 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.703230    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"service-ca-bundle\""
Nov 02 17:29:44.703260 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.703248    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 17:29:44.705214 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.705188    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 17:29:44.705214 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.705198    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 17:29:44.705214 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.705199    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 17:29:44.705214 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.705215    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 17:29:44.705404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.705223    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 17:29:44.705404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.705230    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"trusted-ca-bundle\""
Nov 02 17:29:44.705404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.705227    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 17:29:44.705404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.705250    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 17:29:44.706556 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.706543    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 17:29:44.706636 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.706568    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 17:29:44.706636 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.706571    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 17:29:44.706636 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.706584    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 17:29:44.706636 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.706589    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 17:29:44.707057 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.707044    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 17:29:44.707115 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.707044    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 17:29:44.707115 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.707078    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 17:29:44.707115 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.707084    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 17:29:44.707213 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.707141    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:29:44.707273 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.707259    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 17:29:44.707322 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.707287    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 17:29:44.707322 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.707300    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 17:29:44.707322 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.707300    4456 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:29:44.707449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.707343    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 17:29:44.707449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.707350    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 17:29:44.707449 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:44.707348    4456 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-apiserver-proxy-ip-10-0-141-87.ec2.internal\" already exists" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:44.707449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.707378    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 17:29:44.710367 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.710352    4456 warnings.go:107] "Warning: metadata.name: this is used in the Pod's hostname, which can result in surprising behavior; a DNS label is recommended: [must not contain dots]"
Nov 02 17:29:44.710426 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:44.710384    4456 kubelet.go:3370] "Failed creating a mirror pod" err="pods \"kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal\" already exists" pod="openshift-machine-config-operator/kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal"
Nov 02 17:29:44.734627 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.734607    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-cni-binary-copy\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.734681 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.734639    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:44.734681 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.734655    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-dbus\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:44.734755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.734679    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wsvfn\" (UniqueName: \"kubernetes.io/projected/6767bd29-3165-4e67-81b3-e320e8324034-kube-api-access-wsvfn\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:44.734755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.734749    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnfcz\" (UniqueName: \"kubernetes.io/projected/00b87b36-9ce2-480a-9924-22b33256fe1e-kube-api-access-fnfcz\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:44.734846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.734767    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-kubelet\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.734846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.734793    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-kubelet-config\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:44.734846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.734831    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-slash\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.734978 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.734894    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/367d8d85-a5e8-462d-b1d7-91a01e62b726-agent-certs\") pod \"konnectivity-agent-mxsjq\" (UID: \"367d8d85-a5e8-462d-b1d7-91a01e62b726\") " pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:44.735036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.734970    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-k8s-cni-cncf-io\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.735036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735014    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-cni-bin\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.735124 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735057    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-42fn2\" (UniqueName: \"kubernetes.io/projected/dd733778-7002-4a26-a4e8-4525d8f6bee9-kube-api-access-42fn2\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:44.735124 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735102    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00b87b36-9ce2-480a-9924-22b33256fe1e-host\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:44.735124 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735113    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-cni-binary-copy\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.735315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735207    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00b87b36-9ce2-480a-9924-22b33256fe1e-serviceca\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:44.735315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735227    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce8b7505-62a5-42f1-b432-d77200c21d2d-hosts-file\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:44.735315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735287    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zjwg\" (UniqueName: \"kubernetes.io/projected/ce8b7505-62a5-42f1-b432-d77200c21d2d-kube-api-access-7zjwg\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:44.735315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735294    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"agent-certs\" (UniqueName: \"kubernetes.io/secret/367d8d85-a5e8-462d-b1d7-91a01e62b726-agent-certs\") pod \"konnectivity-agent-mxsjq\" (UID: \"367d8d85-a5e8-462d-b1d7-91a01e62b726\") " pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:44.735443 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735333    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-etc-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.735443 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735426    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-etc-kubernetes\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.735443 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735440    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serviceca\" (UniqueName: \"kubernetes.io/configmap/00b87b36-9ce2-480a-9924-22b33256fe1e-serviceca\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:44.735524 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735460    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-skhvr\" (UniqueName: \"kubernetes.io/projected/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-kube-api-access-skhvr\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.735900 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735879    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:44.735969 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735505    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"original-pull-secret\" (UniqueName: \"kubernetes.io/secret/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-original-pull-secret\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:44.735969 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735937    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-registration-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.735969 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735954    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-etc-selinux\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.735969 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735969    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-var-lib-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.736157 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735983    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.736157 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.735999    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-env-overrides\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.736157 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736037    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6767bd29-3165-4e67-81b3-e320e8324034-iptables-alerter-script\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:44.736157 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736076    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7jnh\" (UniqueName: \"kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh\") pod \"network-check-target-nttxj\" (UID: \"a974c7e3-bf97-4575-bec1-c09fcd018f0d\") " pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:44.736157 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736109    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-service-ca-bundle\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:44.736382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736192    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-trusted-ca-bundle\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:44.736382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736217    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:44.736382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736241    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kd2rw\" (UniqueName: \"kubernetes.io/projected/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-api-access-kd2rw\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:44.736382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736257    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysconfig\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.736382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736278    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-kubernetes\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.736382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736277    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-service-ca-bundle\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:44.736382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736297    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"env-overrides\" (UniqueName: \"kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-env-overrides\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.736382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736277    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"iptables-alerter-script\" (UniqueName: \"kubernetes.io/configmap/6767bd29-3165-4e67-81b3-e320e8324034-iptables-alerter-script\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:44.736382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736314    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-socket-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.736679 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736425    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-run-netns\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.736679 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736458    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.736679 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736527    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4b72e7f-cc4c-4662-a923-c184069aa118-tmp\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.736679 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736579    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-node-log\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.736679 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736635    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tvrzx\" (UniqueName: \"kubernetes.io/projected/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-kube-api-access-tvrzx\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.736856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736724    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/f4b72e7f-cc4c-4662-a923-c184069aa118-tmp\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.736856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736724    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4kfc\" (UniqueName: \"kubernetes.io/projected/83455bdf-65dc-4f66-b943-55835e92b360-kube-api-access-l4kfc\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:44.736856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736803    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysctl-conf\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.736856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736847    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-systemd\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.737045 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736874    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-trusted-ca-bundle\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:44.737045 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736886    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5xrlg\" (UniqueName: \"kubernetes.io/projected/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-kube-api-access-5xrlg\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.737045 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.736985    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-multus-certs\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.737045 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737009    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cni-binary-copy\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.737045 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737034    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-sys-fs\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.737270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737137    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/83455bdf-65dc-4f66-b943-55835e92b360-archives-path\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:44.737270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737167    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysctl-d\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.737270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737178    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-binary-copy\" (UniqueName: \"kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cni-binary-copy\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.737270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737192    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-var-lib-kubelet\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.737270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737230    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6767bd29-3165-4e67-81b3-e320e8324034-host-slash\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:44.737270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737241    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/83455bdf-65dc-4f66-b943-55835e92b360-archives-path\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:44.737270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737254    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-ovn\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.737270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737270    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-cni-netd\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.737537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737288    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.737537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737324    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cnibin\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.737537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737344    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.737537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737360    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4g4b6\" (UniqueName: \"kubernetes.io/projected/758467e6-69e4-4c1f-b344-e1e74497c298-kube-api-access-4g4b6\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.737537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737374    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kfzh6\" (UniqueName: \"kubernetes.io/projected/ba5cbec9-7649-462a-b110-11ebcc5d6d78-kube-api-access-kfzh6\") pod \"ingress-canary-n4r8t\" (UID: \"ba5cbec9-7649-462a-b110-11ebcc5d6d78\") " pod="openshift-ingress-canary/ingress-canary-n4r8t"
Nov 02 17:29:44.737537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737389    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:44.737537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737433    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59eba5d-a500-462e-9ecd-4ef9e553f873-tmp-dir\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:44.737537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737503    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"whereabouts-flatfile-configmap\" (UniqueName: \"kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-whereabouts-flatfile-configmap\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.737537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737513    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f59eba5d-a500-462e-9ecd-4ef9e553f873-tmp-dir\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:44.737843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737564    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovnkube-script-lib\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.737843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737588    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-run\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.737843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737614    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-device-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.737843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737659    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f59eba5d-a500-462e-9ecd-4ef9e553f873-config-volume\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:44.737843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737682    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-systemd-units\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.737843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737709    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-kjmhh\" (UniqueName: \"kubernetes.io/projected/8064da0c-fc5d-493c-bcdf-4327716d4cc2-kube-api-access-kjmhh\") pod \"migrator-66df5d496b-lxxf2\" (UID: \"8064da0c-fc5d-493c-bcdf-4327716d4cc2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2"
Nov 02 17:29:44.737843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737738    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.737843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737772    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-script-lib\" (UniqueName: \"kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovnkube-script-lib\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.737843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737786    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-conf-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.737843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737837    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-system-cni-dir\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.738227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737866    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.738227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737891    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-systemd\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.738227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737904    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-volume\" (UniqueName: \"kubernetes.io/configmap/f59eba5d-a500-462e-9ecd-4ef9e553f873-config-volume\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:44.738227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737923    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cni-sysctl-allowlist\" (UniqueName: \"kubernetes.io/configmap/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cni-sysctl-allowlist\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.738227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737938    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/367d8d85-a5e8-462d-b1d7-91a01e62b726-konnectivity-ca\") pod \"konnectivity-agent-mxsjq\" (UID: \"367d8d85-a5e8-462d-b1d7-91a01e62b726\") " pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:44.738227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737970    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-crio-socket\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:44.738227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.737997    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-socket-dir-parent\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.738227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738050    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-netns\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.738227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738090    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8b7505-62a5-42f1-b432-d77200c21d2d-tmp-dir\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:44.738227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738092    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"konnectivity-ca\" (UniqueName: \"kubernetes.io/configmap/367d8d85-a5e8-462d-b1d7-91a01e62b726-konnectivity-ca\") pod \"konnectivity-agent-mxsjq\" (UID: \"367d8d85-a5e8-462d-b1d7-91a01e62b726\") " pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:44.738227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738117    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.738227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738155    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbbbk\" (UniqueName: \"kubernetes.io/projected/f59eba5d-a500-462e-9ecd-4ef9e553f873-kube-api-access-cbbbk\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:44.738227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738179    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/ce8b7505-62a5-42f1-b432-d77200c21d2d-tmp-dir\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:44.738227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738210    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-cni-bin\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.738227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738232    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovnkube-config\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738253    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovn-node-metrics-cert\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738316    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-cni-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738343    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-cni-multus\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738394    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-log-socket\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738413    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovn-node-metrics-cert\" (UniqueName: \"kubernetes.io/secret/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovn-node-metrics-cert\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738435    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-data-volume\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738461    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738489    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-modprobe-d\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738503    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"data-volume\" (UniqueName: \"kubernetes.io/empty-dir/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-data-volume\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738509    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-host\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738542    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ovnkube-config\" (UniqueName: \"kubernetes.io/configmap/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-ovnkube-config\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738551    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-daemon-config\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738583    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba5cbec9-7649-462a-b110-11ebcc5d6d78-cert\") pod \"ingress-canary-n4r8t\" (UID: \"ba5cbec9-7649-462a-b110-11ebcc5d6d78\") " pod="openshift-ingress-canary/ingress-canary-n4r8t"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738599    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-sys\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738626    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-lib-modules\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738644    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-tuned\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738705    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-tuned\" (UniqueName: \"kubernetes.io/empty-dir/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-tuned\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738698    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f59q8\" (UniqueName: \"kubernetes.io/projected/f4b72e7f-cc4c-4662-a923-c184069aa118-kube-api-access-f59q8\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.738734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738735    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/ba5cbec9-7649-462a-b110-11ebcc5d6d78-cert\") pod \"ingress-canary-n4r8t\" (UID: \"ba5cbec9-7649-462a-b110-11ebcc5d6d78\") " pod="openshift-ingress-canary/ingress-canary-n4r8t"
Nov 02 17:29:44.739309 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738748    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-hostroot\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.739309 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738777    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-kubelet\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.739309 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738803    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-system-cni-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.739309 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738848    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-daemon-config\" (UniqueName: \"kubernetes.io/configmap/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-daemon-config\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.739309 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.738871    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-cnibin\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.776838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.776793    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7jnh\" (UniqueName: \"kubernetes.io/projected/a974c7e3-bf97-4575-bec1-c09fcd018f0d-kube-api-access-d7jnh\") pod \"network-check-target-nttxj\" (UID: \"a974c7e3-bf97-4575-bec1-c09fcd018f0d\") " pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:44.777271 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.777229    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-skhvr\" (UniqueName: \"kubernetes.io/projected/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-kube-api-access-skhvr\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.777364 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.777341    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4kfc\" (UniqueName: \"kubernetes.io/projected/83455bdf-65dc-4f66-b943-55835e92b360-kube-api-access-l4kfc\") pod \"periodic-gathering-wphg6-wf57n\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") " pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:29:44.777470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.777456    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbbbk\" (UniqueName: \"kubernetes.io/projected/f59eba5d-a500-462e-9ecd-4ef9e553f873-kube-api-access-cbbbk\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:44.789589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.789570    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zjwg\" (UniqueName: \"kubernetes.io/projected/ce8b7505-62a5-42f1-b432-d77200c21d2d-kube-api-access-7zjwg\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:44.789807 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.789786    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kd2rw\" (UniqueName: \"kubernetes.io/projected/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-api-access-kd2rw\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:44.789886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.789862    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kfzh6\" (UniqueName: \"kubernetes.io/projected/ba5cbec9-7649-462a-b110-11ebcc5d6d78-kube-api-access-kfzh6\") pod \"ingress-canary-n4r8t\" (UID: \"ba5cbec9-7649-462a-b110-11ebcc5d6d78\") " pod="openshift-ingress-canary/ingress-canary-n4r8t"
Nov 02 17:29:44.794962 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.794945    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wsvfn\" (UniqueName: \"kubernetes.io/projected/6767bd29-3165-4e67-81b3-e320e8324034-kube-api-access-wsvfn\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:44.796248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.796230    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tvrzx\" (UniqueName: \"kubernetes.io/projected/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-kube-api-access-tvrzx\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.796654 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.796634    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5xrlg\" (UniqueName: \"kubernetes.io/projected/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-kube-api-access-5xrlg\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.796952 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.796934    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-kjmhh\" (UniqueName: \"kubernetes.io/projected/8064da0c-fc5d-493c-bcdf-4327716d4cc2-kube-api-access-kjmhh\") pod \"migrator-66df5d496b-lxxf2\" (UID: \"8064da0c-fc5d-493c-bcdf-4327716d4cc2\") " pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2"
Nov 02 17:29:44.797107 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.797090    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f59q8\" (UniqueName: \"kubernetes.io/projected/f4b72e7f-cc4c-4662-a923-c184069aa118-kube-api-access-f59q8\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.797276 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.797261    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnfcz\" (UniqueName: \"kubernetes.io/projected/00b87b36-9ce2-480a-9924-22b33256fe1e-kube-api-access-fnfcz\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:44.799244 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.799227    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4g4b6\" (UniqueName: \"kubernetes.io/projected/758467e6-69e4-4c1f-b344-e1e74497c298-kube-api-access-4g4b6\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.801826 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.801797    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-42fn2\" (UniqueName: \"kubernetes.io/projected/dd733778-7002-4a26-a4e8-4525d8f6bee9-kube-api-access-42fn2\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:44.840021 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.839967    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:44.840021 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.839993    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysconfig\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.840021 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840008    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-kubernetes\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.840179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840023    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-socket-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.840179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840046    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-run-netns\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.840179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840060    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.840179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840087    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysconfig\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysconfig\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.840179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840109    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-run-netns\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.840179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840076    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-node-log\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.840179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840125    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-run-ovn-kubernetes\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.840179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840136    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-log\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-node-log\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.840179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840121    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"socket-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-socket-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.840179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840091    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-kubernetes\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.840179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840148    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysctl-conf\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:44.840183    4456 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840191    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-systemd\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840242    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-conf\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysctl-conf\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840249    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-systemd\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-systemd\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840252    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-multus-certs\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:44.840274    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls podName:da9f1b66-0e1b-4c7e-a947-a897c00ccc19 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:45.340251214 +0000 UTC m=+1.831314383 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6bfcf" (UID: "da9f1b66-0e1b-4c7e-a947-a897c00ccc19") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840301    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-multus-certs\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-multus-certs\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840341    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-sys-fs\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840363    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysctl-d\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840378    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-var-lib-kubelet\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840392    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6767bd29-3165-4e67-81b3-e320e8324034-host-slash\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840407    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-ovn\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840422    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys-fs\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-sys-fs\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840446    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-var-lib-kubelet\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840436    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-cni-netd\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840466    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/6767bd29-3165-4e67-81b3-e320e8324034-host-slash\") pod \"iptables-alerter-dlp6w\" (UID: \"6767bd29-3165-4e67-81b3-e320e8324034\") " pod="openshift-network-operator/iptables-alerter-dlp6w"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840492    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-netd\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-cni-netd\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840497    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-ovn\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-ovn\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840512    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-sysctl-d\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-sysctl-d\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840534    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840569    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cnibin\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840590    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-networks-ovn-kubernetes\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-var-lib-cni-networks-ovn-kubernetes\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840594    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840619    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-cnibin\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840621    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-run\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840644    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-device-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840672    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-systemd-units\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:44.840687    4456 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840712    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-run\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840718    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"systemd-units\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-systemd-units\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840715    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-conf-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840714    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"device-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-device-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:44.840741    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls podName:f59eba5d-a500-462e-9ecd-4ef9e553f873 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:45.340724955 +0000 UTC m=+1.831788108 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls") pod "dns-default-zmrdt" (UID: "f59eba5d-a500-462e-9ecd-4ef9e553f873") : secret "dns-default-metrics-tls" not found
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840764    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-system-cni-dir\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.840798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840738    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-conf-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-conf-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840828    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-system-cni-dir\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.840780    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841116    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-systemd\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841168    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-crio-socket\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841199    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-socket-dir-parent\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841227    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-netns\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841225    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tuning-conf-dir\" (UniqueName: \"kubernetes.io/host-path/c2d7ba52-9a2b-4871-aca0-3c4e1934c572-tuning-conf-dir\") pod \"multus-additional-cni-plugins-ngkbz\" (UID: \"c2d7ba52-9a2b-4871-aca0-3c4e1934c572\") " pod="openshift-multus/multus-additional-cni-plugins-ngkbz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841286    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"crio-socket\" (UniqueName: \"kubernetes.io/host-path/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-crio-socket\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841304    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-socket-dir-parent\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-socket-dir-parent\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841330    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841356    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-netns\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-netns\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841361    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-cni-bin\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841414    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-cni-bin\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-cni-bin\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841452    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-cni-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841486    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-cni-multus\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841518    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-log-socket\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841580    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-multus\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-cni-multus\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841619    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841651    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-modprobe-d\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841675    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-host\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841714    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-sys\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841746    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-lib-modules\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841784    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-hostroot\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841836    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-kubelet\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841869    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-system-cni-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841894    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-cnibin\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841929    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841960    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-dbus\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.841995    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-kubelet\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842036    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-kubelet-config\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842069    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-slash\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842102    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-k8s-cni-cncf-io\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842134    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-cni-bin\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842167    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00b87b36-9ce2-480a-9924-22b33256fe1e-host\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842194    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce8b7505-62a5-42f1-b432-d77200c21d2d-hosts-file\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842226    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-etc-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842266    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-etc-kubernetes\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842299    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-registration-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.842323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842330    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-etc-selinux\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842355    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-var-lib-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842387    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842489    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842536    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"log-socket\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-log-socket\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842578    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"run-systemd\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-run-systemd\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842630    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-kubelet-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.842684    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"multus-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-multus-cni-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843102    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-kubelet\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-kubelet\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843286    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-rbac-proxy-cm\" (UniqueName: \"kubernetes.io/configmap/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-kube-rbac-proxy-cm\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843368    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kubelet-config\" (UniqueName: \"kubernetes.io/host-path/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-kubelet-config\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843422    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-modprobe-d\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-etc-modprobe-d\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843435    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-host\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843479    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-etc-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843489    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host\" (UniqueName: \"kubernetes.io/host-path/00b87b36-9ce2-480a-9924-22b33256fe1e-host\") pod \"node-ca-lhwgm\" (UID: \"00b87b36-9ce2-480a-9924-22b33256fe1e\") " pod="openshift-image-registry/node-ca-lhwgm"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843509    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-sys\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843534    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"var-lib-openvswitch\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-var-lib-openvswitch\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843543    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-kubernetes\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-etc-kubernetes\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843441    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hosts-file\" (UniqueName: \"kubernetes.io/host-path/ce8b7505-62a5-42f1-b432-d77200c21d2d-hosts-file\") pod \"node-resolver-hhsrb\" (UID: \"ce8b7505-62a5-42f1-b432-d77200c21d2d\") " pod="openshift-dns/node-resolver-hhsrb"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843586    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-kubelet\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-kubelet\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843601    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dbus\" (UniqueName: \"kubernetes.io/host-path/8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d-dbus\") pod \"global-pull-secret-syncer-mmhd7\" (UID: \"8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d\") " pod="kube-system/global-pull-secret-syncer-mmhd7"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843671    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cnibin\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-cnibin\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843712    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-var-lib-cni-bin\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-var-lib-cni-bin\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843756    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"system-cni-dir\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-system-cni-dir\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843858    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"etc-selinux\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-etc-selinux\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.843956    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/f4b72e7f-cc4c-4662-a923-c184069aa118-lib-modules\") pod \"tuned-n9fk9\" (UID: \"f4b72e7f-cc4c-4662-a923-c184069aa118\") " pod="openshift-cluster-node-tuning-operator/tuned-n9fk9"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.844005    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"hostroot\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-hostroot\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.844053    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-run-k8s-cni-cncf-io\" (UniqueName: \"kubernetes.io/host-path/f26fab1e-e6d4-4caa-bef6-81a09d78c0de-host-run-k8s-cni-cncf-io\") pod \"multus-7qzkz\" (UID: \"f26fab1e-e6d4-4caa-bef6-81a09d78c0de\") " pod="openshift-multus/multus-7qzkz"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.844092    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"host-slash\" (UniqueName: \"kubernetes.io/host-path/c86c4f40-2978-40f0-bfc1-c3cfffca0d8c-host-slash\") pod \"ovnkube-node-44l94\" (UID: \"c86c4f40-2978-40f0-bfc1-c3cfffca0d8c\") " pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.844146    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registration-dir\" (UniqueName: \"kubernetes.io/host-path/758467e6-69e4-4c1f-b344-e1e74497c298-registration-dir\") pod \"aws-ebs-csi-driver-node-6bb2n\" (UID: \"758467e6-69e4-4c1f-b344-e1e74497c298\") " pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:44.844232    4456 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:29:44.844989 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:44.844351    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs podName:dd733778-7002-4a26-a4e8-4525d8f6bee9 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:45.344336431 +0000 UTC m=+1.835399570 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs") pod "network-metrics-daemon-tk8ht" (UID: "dd733778-7002-4a26-a4e8-4525d8f6bee9") : secret "metrics-daemon-secret" not found
Nov 02 17:29:44.948944 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:44.948905    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:45.119878 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:45.119749    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:45.312552 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:45.312503    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:45.346621 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:45.346590    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:45.346692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:45.346643    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:45.346724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:45.346686    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:45.346772 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:45.346747    4456 secret.go:189] Couldn't get secret openshift-dns/dns-default-metrics-tls: secret "dns-default-metrics-tls" not found
Nov 02 17:29:45.346772 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:45.346769    4456 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:45.346864 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:45.346795    4456 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:29:45.346864 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:45.346846    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls podName:da9f1b66-0e1b-4c7e-a947-a897c00ccc19 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.346808374 +0000 UTC m=+2.837871515 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6bfcf" (UID: "da9f1b66-0e1b-4c7e-a947-a897c00ccc19") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:45.346864 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:45.346862    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls podName:f59eba5d-a500-462e-9ecd-4ef9e553f873 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.346854471 +0000 UTC m=+2.837917609 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-tls" (UniqueName: "kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls") pod "dns-default-zmrdt" (UID: "f59eba5d-a500-462e-9ecd-4ef9e553f873") : secret "dns-default-metrics-tls" not found
Nov 02 17:29:45.346971 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:45.346882    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs podName:dd733778-7002-4a26-a4e8-4525d8f6bee9 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:46.346868534 +0000 UTC m=+2.837931673 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs") pod "network-metrics-daemon-tk8ht" (UID: "dd733778-7002-4a26-a4e8-4525d8f6bee9") : secret "metrics-daemon-secret" not found
Nov 02 17:29:45.565879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:45.565849    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lhwgm_00b87b36-9ce2-480a-9924-22b33256fe1e/node-ca/0.log"
Nov 02 17:29:45.589968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:45.589938    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:45.765900 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:45.765857    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:46.020537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:46.020509    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:46.030803 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:46.030776    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:46.353146 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:46.353063    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:46.353146 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:46.353105    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:46.353146 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:46.353134    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:46.353362 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:46.353215    4456 secret.go:189] Couldn't get secret openshift-multus/metrics-daemon-secret: secret "metrics-daemon-secret" not found
Nov 02 17:29:46.353362 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:46.353235    4456 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:46.353362 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:46.353279    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs podName:dd733778-7002-4a26-a4e8-4525d8f6bee9 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:48.353265154 +0000 UTC m=+4.844328309 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "metrics-certs" (UniqueName: "kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs") pod "network-metrics-daemon-tk8ht" (UID: "dd733778-7002-4a26-a4e8-4525d8f6bee9") : secret "metrics-daemon-secret" not found
Nov 02 17:29:46.353362 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:46.353293    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls podName:da9f1b66-0e1b-4c7e-a947-a897c00ccc19 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:48.353286985 +0000 UTC m=+4.844350124 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6bfcf" (UID: "da9f1b66-0e1b-4c7e-a947-a897c00ccc19") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:46.353515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:46.353485    4456 swap_util.go:78] "error creating dir to test if tmpfs noswap is enabled. Assuming not supported" mount path="" error="stat /var/lib/kubelet/plugins/kubernetes.io/empty-dir: no such file or directory"
Nov 02 17:29:46.355614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:46.355599    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-tls\" (UniqueName: \"kubernetes.io/secret/f59eba5d-a500-462e-9ecd-4ef9e553f873-metrics-tls\") pod \"dns-default-zmrdt\" (UID: \"f59eba5d-a500-462e-9ecd-4ef9e553f873\") " pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:46.357790 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:46.357769    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress-canary_ingress-canary-n4r8t_ba5cbec9-7649-462a-b110-11ebcc5d6d78/serve-healthcheck-canary/0.log"
Nov 02 17:29:46.427646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:46.427620    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:46.727197 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:46.727161    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-dns/dns-default-zmrdt"]
Nov 02 17:29:46.732997 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:46.732971    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf59eba5d_a500_462e_9ecd_4ef9e553f873.slice/crio-9baaab8c93f87f917ea2357edc3f6a1082f29847963c828705ab2521afbf8dcd WatchSource:0}: Error finding container 9baaab8c93f87f917ea2357edc3f6a1082f29847963c828705ab2521afbf8dcd: Status 404 returned error can't find the container with id 9baaab8c93f87f917ea2357edc3f6a1082f29847963c828705ab2521afbf8dcd
Nov 02 17:29:46.735272 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:46.735249    4456 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:29:46.750964 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:46.750944    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-lxxf2_8064da0c-fc5d-493c-bcdf-4327716d4cc2/migrator/0.log"
Nov 02 17:29:46.849837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:46.849787    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 17:29:46.900762 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:46.900718    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:47.114158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:47.114076    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:47.293758 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:47.293723    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:47.448174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:47.448138    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:47.592090 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:47.592061    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:47.653749 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:47.653707    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmrdt" event={"ID":"f59eba5d-a500-462e-9ecd-4ef9e553f873","Type":"ContainerStarted","Data":"9baaab8c93f87f917ea2357edc3f6a1082f29847963c828705ab2521afbf8dcd"}
Nov 02 17:29:47.802324 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:47.802232    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:48.090264 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:48.090180    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:29:48.371681 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:48.371606    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:48.371681 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:48.371666    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:48.371930 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:48.371774    4456 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:48.371930 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:48.371871    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls podName:da9f1b66-0e1b-4c7e-a947-a897c00ccc19 nodeName:}" failed. No retries permitted until 2026-11-02 17:29:52.37185117 +0000 UTC m=+8.862914323 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6bfcf" (UID: "da9f1b66-0e1b-4c7e-a947-a897c00ccc19") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:48.373965 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:48.373940    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/dd733778-7002-4a26-a4e8-4525d8f6bee9-metrics-certs\") pod \"network-metrics-daemon-tk8ht\" (UID: \"dd733778-7002-4a26-a4e8-4525d8f6bee9\") " pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:48.525491 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:48.525469    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-multus/network-metrics-daemon-tk8ht"
Nov 02 17:29:48.658377 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:48.658289    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmrdt" event={"ID":"f59eba5d-a500-462e-9ecd-4ef9e553f873","Type":"ContainerStarted","Data":"a39d732409ae43b63786134e37724b226b23d17b80d0af28eea5639155b8bd85"}
Nov 02 17:29:48.658377 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:48.658351    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-dns/dns-default-zmrdt" event={"ID":"f59eba5d-a500-462e-9ecd-4ef9e553f873","Type":"ContainerStarted","Data":"350a47fa5415c9e35aaa82716bd418175a20de648fe6dd1f5b2a653ec15f1857"}
Nov 02 17:29:48.658557 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:48.658431    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:48.825228 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:48.825197    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-multus/network-metrics-daemon-tk8ht"]
Nov 02 17:29:48.828268 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:29:48.828245    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddd733778_7002_4a26_a4e8_4525d8f6bee9.slice/crio-3b09f7acede9ca9322f99cc75696f683edb9b8790d6959617d5dd88f1e5c9eb8 WatchSource:0}: Error finding container 3b09f7acede9ca9322f99cc75696f683edb9b8790d6959617d5dd88f1e5c9eb8: Status 404 returned error can't find the container with id 3b09f7acede9ca9322f99cc75696f683edb9b8790d6959617d5dd88f1e5c9eb8
Nov 02 17:29:48.900797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:48.900760    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:49.022886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:49.022854    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:29:49.023094 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:49.023079    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 17:29:49.049755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:49.049729    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:49.222488 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:49.222443    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:49.396630 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:49.396507    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:49.662079 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:49.662004    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:49.662269 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:49.662240    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tk8ht" event={"ID":"dd733778-7002-4a26-a4e8-4525d8f6bee9","Type":"ContainerStarted","Data":"3b09f7acede9ca9322f99cc75696f683edb9b8790d6959617d5dd88f1e5c9eb8"}
Nov 02 17:29:49.861041 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:49.861003    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:50.460015 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:50.459974    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:50.668830 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:50.668781    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tk8ht" event={"ID":"dd733778-7002-4a26-a4e8-4525d8f6bee9","Type":"ContainerStarted","Data":"b26050fbdfe0cc383fc244079b41676f90b91a7dae6c13a76f74e5e6eb8d4e5f"}
Nov 02 17:29:50.668979 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:50.668890    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-multus/network-metrics-daemon-tk8ht" event={"ID":"dd733778-7002-4a26-a4e8-4525d8f6bee9","Type":"ContainerStarted","Data":"3e1fb199fc5d6f58caf0f830a6522b0d40d6a62b85b839bff447bb069af17f0d"}
Nov 02 17:29:50.918976 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:50.918891    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:51.113883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:51.113850    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:51.293144 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:51.293107    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:51.497792 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:51.497760    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:51.649535 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:51.649458    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:51.793892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:51.793858    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:52.397196 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:52.397162    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:29:52.397581 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:52.397295    4456 secret.go:189] Couldn't get secret openshift-insights/insights-runtime-extractor-tls: secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:52.397581 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:29:52.397359    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls podName:da9f1b66-0e1b-4c7e-a947-a897c00ccc19 nodeName:}" failed. No retries permitted until 2026-11-02 17:30:00.397341733 +0000 UTC m=+16.888404872 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "insights-runtime-extractor-tls" (UniqueName: "kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls") pod "insights-runtime-extractor-6bfcf" (UID: "da9f1b66-0e1b-4c7e-a947-a897c00ccc19") : secret "insights-runtime-extractor-tls" not found
Nov 02 17:29:52.693390 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:52.693352    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:53.144115 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:53.144036    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:53.355019 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:53.354983    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:53.519251 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:53.519221    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:53.906189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:53.906109    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:54.707672 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:54.707637    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:55.115122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:55.115048    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:55.267452 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:55.267418    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:55.414340 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:55.414263    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:55.555434 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:55.555398    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:55.924185 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:55.924151    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:56.125768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:56.125737    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:56.385869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:56.385836    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:56.556675 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:56.556638    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:57.229320 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:57.229294    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:57.383168 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:57.383138    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:57.555868 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:57.555792    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:58.664938 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:58.664911    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zmrdt"
Nov 02 17:29:59.819112 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:59.819075    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:29:59.969411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:29:59.969374    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:00.449167 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:00.449134    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:30:00.451675 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:00.451654    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"insights-runtime-extractor-tls\" (UniqueName: \"kubernetes.io/secret/da9f1b66-0e1b-4c7e-a947-a897c00ccc19-insights-runtime-extractor-tls\") pod \"insights-runtime-extractor-6bfcf\" (UID: \"da9f1b66-0e1b-4c7e-a947-a897c00ccc19\") " pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:30:00.529726 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:00.529697    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-insights/insights-runtime-extractor-6bfcf"
Nov 02 17:30:00.850836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:00.850728    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-insights/insights-runtime-extractor-6bfcf"]
Nov 02 17:30:00.854415 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:30:00.854384    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda9f1b66_0e1b_4c7e_a947_a897c00ccc19.slice/crio-137796149fd72d9e2ebe90b225822de45acbe2e1e0820da14a6da20aa2c3f28d WatchSource:0}: Error finding container 137796149fd72d9e2ebe90b225822de45acbe2e1e0820da14a6da20aa2c3f28d: Status 404 returned error can't find the container with id 137796149fd72d9e2ebe90b225822de45acbe2e1e0820da14a6da20aa2c3f28d
Nov 02 17:30:00.908573 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:00.908545    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:01.057532 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.057497    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:01.391013 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.390984    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"]
Nov 02 17:30:01.393207 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.393186    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 17:30:01.410025 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.409998    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-console/downloads-6dd789c8b7-pgmzg"]
Nov 02 17:30:01.412025 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.412006    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-pgmzg"
Nov 02 17:30:01.422354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.422333    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 02 17:30:01.422486 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.422344    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mhkk4\""
Nov 02 17:30:01.433969 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.433946    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"]
Nov 02 17:30:01.434799 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.434776    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 17:30:01.434902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.434837    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 17:30:01.442829 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.442793    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-pgmzg"]
Nov 02 17:30:01.489980 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.489955    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-image-registry/image-registry-59767d8795-7vwkt"]
Nov 02 17:30:01.493114 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.492769    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.529869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.529847    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 17:30:01.530097 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.530056    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-gclvz\""
Nov 02 17:30:01.530203 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.530092    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 17:30:01.530203 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.530115    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 17:30:01.530203 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.530185    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 17:30:01.530203 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.530199    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mnv5k\""
Nov 02 17:30:01.530964 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.530944    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59767d8795-7vwkt"]
Nov 02 17:30:01.556680 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.556658    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7s4l4\" (UniqueName: \"kubernetes.io/projected/6e78d6c7-c325-48f2-af90-804dade84d8f-kube-api-access-7s4l4\") pod \"downloads-6dd789c8b7-pgmzg\" (UID: \"6e78d6c7-c325-48f2-af90-804dade84d8f\") " pod="openshift-console/downloads-6dd789c8b7-pgmzg"
Nov 02 17:30:01.556774 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.556699    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cae854c8-88f7-4201-b5d7-4e7c59fdf833-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-q2mbj\" (UID: \"cae854c8-88f7-4201-b5d7-4e7c59fdf833\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 17:30:01.556774 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.556766    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e78d6c7-c325-48f2-af90-804dade84d8f-tmp\") pod \"downloads-6dd789c8b7-pgmzg\" (UID: \"6e78d6c7-c325-48f2-af90-804dade84d8f\") " pod="openshift-console/downloads-6dd789c8b7-pgmzg"
Nov 02 17:30:01.657444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.657359    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0db36d92-4712-4338-958a-ebabf2c137ba-trusted-ca\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.657444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.657389    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0db36d92-4712-4338-958a-ebabf2c137ba-installation-pull-secrets\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.657444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.657419    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0db36d92-4712-4338-958a-ebabf2c137ba-registry-certificates\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.657444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.657440    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cae854c8-88f7-4201-b5d7-4e7c59fdf833-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-q2mbj\" (UID: \"cae854c8-88f7-4201-b5d7-4e7c59fdf833\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 17:30:01.657842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.657616    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e78d6c7-c325-48f2-af90-804dade84d8f-tmp\") pod \"downloads-6dd789c8b7-pgmzg\" (UID: \"6e78d6c7-c325-48f2-af90-804dade84d8f\") " pod="openshift-console/downloads-6dd789c8b7-pgmzg"
Nov 02 17:30:01.657842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.657678    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0db36d92-4712-4338-958a-ebabf2c137ba-registry-tls\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.657842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.657756    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0db36d92-4712-4338-958a-ebabf2c137ba-image-registry-private-configuration\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.657842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.657789    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0db36d92-4712-4338-958a-ebabf2c137ba-ca-trust-extracted\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.657842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.657829    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0db36d92-4712-4338-958a-ebabf2c137ba-bound-sa-token\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.658078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.657862    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hjqqd\" (UniqueName: \"kubernetes.io/projected/0db36d92-4712-4338-958a-ebabf2c137ba-kube-api-access-hjqqd\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.658078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.657911    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7s4l4\" (UniqueName: \"kubernetes.io/projected/6e78d6c7-c325-48f2-af90-804dade84d8f-kube-api-access-7s4l4\") pod \"downloads-6dd789c8b7-pgmzg\" (UID: \"6e78d6c7-c325-48f2-af90-804dade84d8f\") " pod="openshift-console/downloads-6dd789c8b7-pgmzg"
Nov 02 17:30:01.658078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.658000    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e78d6c7-c325-48f2-af90-804dade84d8f-tmp\") pod \"downloads-6dd789c8b7-pgmzg\" (UID: \"6e78d6c7-c325-48f2-af90-804dade84d8f\") " pod="openshift-console/downloads-6dd789c8b7-pgmzg"
Nov 02 17:30:01.660174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.660152    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certificates\" (UniqueName: \"kubernetes.io/secret/cae854c8-88f7-4201-b5d7-4e7c59fdf833-tls-certificates\") pod \"prometheus-operator-admission-webhook-6489d8bf89-q2mbj\" (UID: \"cae854c8-88f7-4201-b5d7-4e7c59fdf833\") " pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 17:30:01.688610 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.688589    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7s4l4\" (UniqueName: \"kubernetes.io/projected/6e78d6c7-c325-48f2-af90-804dade84d8f-kube-api-access-7s4l4\") pod \"downloads-6dd789c8b7-pgmzg\" (UID: \"6e78d6c7-c325-48f2-af90-804dade84d8f\") " pod="openshift-console/downloads-6dd789c8b7-pgmzg"
Nov 02 17:30:01.700737 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.700711    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:01.704719 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.704704    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 17:30:01.707207 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.707185    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6bfcf" event={"ID":"da9f1b66-0e1b-4c7e-a947-a897c00ccc19","Type":"ContainerStarted","Data":"cab46421ec6f270ec6cf6afdc360037817043b96dbc618da9d1220328d936284"}
Nov 02 17:30:01.707288 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.707239    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6bfcf" event={"ID":"da9f1b66-0e1b-4c7e-a947-a897c00ccc19","Type":"ContainerStarted","Data":"accae25fcadb666a2de6b6b0629204a1956cafe875a63f53887a4ef68d269613"}
Nov 02 17:30:01.707288 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.707261    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6bfcf" event={"ID":"da9f1b66-0e1b-4c7e-a947-a897c00ccc19","Type":"ContainerStarted","Data":"137796149fd72d9e2ebe90b225822de45acbe2e1e0820da14a6da20aa2c3f28d"}
Nov 02 17:30:01.723350 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.723326    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-pgmzg"
Nov 02 17:30:01.763872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.758178    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0db36d92-4712-4338-958a-ebabf2c137ba-registry-certificates\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.763872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.758218    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0db36d92-4712-4338-958a-ebabf2c137ba-registry-tls\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.763872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.758246    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0db36d92-4712-4338-958a-ebabf2c137ba-image-registry-private-configuration\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.763872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.758272    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0db36d92-4712-4338-958a-ebabf2c137ba-ca-trust-extracted\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.763872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.758288    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0db36d92-4712-4338-958a-ebabf2c137ba-bound-sa-token\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.763872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.758302    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hjqqd\" (UniqueName: \"kubernetes.io/projected/0db36d92-4712-4338-958a-ebabf2c137ba-kube-api-access-hjqqd\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.763872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.758321    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0db36d92-4712-4338-958a-ebabf2c137ba-trusted-ca\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.763872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.758338    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0db36d92-4712-4338-958a-ebabf2c137ba-installation-pull-secrets\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.763872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.758965    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"ca-trust-extracted\" (UniqueName: \"kubernetes.io/empty-dir/0db36d92-4712-4338-958a-ebabf2c137ba-ca-trust-extracted\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.763872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.759166    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-certificates\" (UniqueName: \"kubernetes.io/configmap/0db36d92-4712-4338-958a-ebabf2c137ba-registry-certificates\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.763872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.759775    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"trusted-ca\" (UniqueName: \"kubernetes.io/configmap/0db36d92-4712-4338-958a-ebabf2c137ba-trusted-ca\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.763872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.761630    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"image-registry-private-configuration\" (UniqueName: \"kubernetes.io/secret/0db36d92-4712-4338-958a-ebabf2c137ba-image-registry-private-configuration\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.767478 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.767454    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"installation-pull-secrets\" (UniqueName: \"kubernetes.io/secret/0db36d92-4712-4338-958a-ebabf2c137ba-installation-pull-secrets\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.768726 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.768703    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"registry-tls\" (UniqueName: \"kubernetes.io/projected/0db36d92-4712-4338-958a-ebabf2c137ba-registry-tls\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.786691 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.786665    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/0db36d92-4712-4338-958a-ebabf2c137ba-bound-sa-token\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.786775 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.786718    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hjqqd\" (UniqueName: \"kubernetes.io/projected/0db36d92-4712-4338-958a-ebabf2c137ba-kube-api-access-hjqqd\") pod \"image-registry-59767d8795-7vwkt\" (UID: \"0db36d92-4712-4338-958a-ebabf2c137ba\") " pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.820756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.820725    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:01.950075 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:01.950038    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:02.005979 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:02.005953    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"]
Nov 02 17:30:02.014681 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:30:02.014652    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcae854c8_88f7_4201_b5d7_4e7c59fdf833.slice/crio-687448bb452bf909f13559f41a0dad969e7b5badd1a742b30388a789a867d1de WatchSource:0}: Error finding container 687448bb452bf909f13559f41a0dad969e7b5badd1a742b30388a789a867d1de: Status 404 returned error can't find the container with id 687448bb452bf909f13559f41a0dad969e7b5badd1a742b30388a789a867d1de
Nov 02 17:30:02.032964 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:02.032942    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-console/downloads-6dd789c8b7-pgmzg"]
Nov 02 17:30:02.039022 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:30:02.038992    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e78d6c7_c325_48f2_af90_804dade84d8f.slice/crio-0fc277289305105311582e26f2bdf721ba404bf132d41beaef4f4aa7518b9ace WatchSource:0}: Error finding container 0fc277289305105311582e26f2bdf721ba404bf132d41beaef4f4aa7518b9ace: Status 404 returned error can't find the container with id 0fc277289305105311582e26f2bdf721ba404bf132d41beaef4f4aa7518b9ace
Nov 02 17:30:02.121658 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:02.121625    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-image-registry/image-registry-59767d8795-7vwkt"]
Nov 02 17:30:02.124678 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:30:02.124653    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0db36d92_4712_4338_958a_ebabf2c137ba.slice/crio-f6f7b8c32295c897f3b29a9213238cc457479edf6b2115de0aa39e12cdf73a81 WatchSource:0}: Error finding container f6f7b8c32295c897f3b29a9213238cc457479edf6b2115de0aa39e12cdf73a81: Status 404 returned error can't find the container with id f6f7b8c32295c897f3b29a9213238cc457479edf6b2115de0aa39e12cdf73a81
Nov 02 17:30:02.712440 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:02.712400    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-pgmzg" event={"ID":"6e78d6c7-c325-48f2-af90-804dade84d8f","Type":"ContainerStarted","Data":"0fc277289305105311582e26f2bdf721ba404bf132d41beaef4f4aa7518b9ace"}
Nov 02 17:30:02.714725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:02.714688    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" event={"ID":"0db36d92-4712-4338-958a-ebabf2c137ba","Type":"ContainerStarted","Data":"2501b584d5ee7323484f50ac5f32dd50c7fef23cf98bd9bf3120550c1d0985af"}
Nov 02 17:30:02.714883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:02.714739    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" event={"ID":"0db36d92-4712-4338-958a-ebabf2c137ba","Type":"ContainerStarted","Data":"f6f7b8c32295c897f3b29a9213238cc457479edf6b2115de0aa39e12cdf73a81"}
Nov 02 17:30:02.715503 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:02.715474    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:02.717156 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:02.717131    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" event={"ID":"cae854c8-88f7-4201-b5d7-4e7c59fdf833","Type":"ContainerStarted","Data":"687448bb452bf909f13559f41a0dad969e7b5badd1a742b30388a789a867d1de"}
Nov 02 17:30:02.762361 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:02.762305    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" podStartSLOduration=1.7622860710000001 podStartE2EDuration="1.762286071s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:02.762228337 +0000 UTC m=+19.253291502" watchObservedRunningTime="2026-11-02 17:30:02.762286071 +0000 UTC m=+19.253349285"
Nov 02 17:30:03.462246 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:03.462221    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ovn-kubernetes/ovnkube-node-44l94"
Nov 02 17:30:03.723257 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:03.723220    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/insights-runtime-extractor-6bfcf" event={"ID":"da9f1b66-0e1b-4c7e-a947-a897c00ccc19","Type":"ContainerStarted","Data":"74a898a99da2aeb1f4e0d0d11c0e63e0285e36c1d51b352f6436433839009bbd"}
Nov 02 17:30:03.725675 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:03.725641    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" event={"ID":"cae854c8-88f7-4201-b5d7-4e7c59fdf833","Type":"ContainerStarted","Data":"31ea127575345103ee79ddb9337e7680f26be2c47d1f8b1c1b65da3c280b1211"}
Nov 02 17:30:03.726008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:03.725988    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 17:30:03.732942 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:03.732923    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 17:30:03.811963 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:03.811778    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podStartSLOduration=1.54721419 podStartE2EDuration="2.811760967s" totalImagesPullingTime="1.264546777s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:03.811054619 +0000 UTC m=+20.302117805" watchObservedRunningTime="2026-11-02 17:30:03.811760967 +0000 UTC m=+20.302824126"
Nov 02 17:30:04.498677 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:04.498643    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:04.656338 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:04.656302    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:04.824238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:04.824152    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:05.210794 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:05.210754    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:07.464057 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:07.464021    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:10.102039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.102000    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:10.272187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.272146    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:10.455334 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.455305    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"]
Nov 02 17:30:10.458422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.458393    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"
Nov 02 17:30:10.480447 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.480424    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 17:30:10.480565 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.480525    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 17:30:10.493144 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.493117    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rdt6t\""
Nov 02 17:30:10.493299 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.493200    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 17:30:10.493299 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.493229    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 17:30:10.495611 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.495591    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 17:30:10.507067 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.507040    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"]
Nov 02 17:30:10.517436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.517407    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:10.526479 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.526450    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/node-exporter-r45c5"]
Nov 02 17:30:10.528806 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.528773    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.579728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.579700    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 17:30:10.583547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.583525    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 17:30:10.583973 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.583958    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 17:30:10.584050 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.584033    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sn4p8\""
Nov 02 17:30:10.630267 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.630220    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c89d394-c878-4069-b3a9-cd0abc822fec-node-exporter-tls\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.630476 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.630293    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c89d394-c878-4069-b3a9-cd0abc822fec-metrics-client-ca\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.630476 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.630415    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9257d59b-6aaf-423e-b7ed-2e64dbdfa535-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-m89vg\" (UID: \"9257d59b-6aaf-423e-b7ed-2e64dbdfa535\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"
Nov 02 17:30:10.630476 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.630446    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9257d59b-6aaf-423e-b7ed-2e64dbdfa535-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-m89vg\" (UID: \"9257d59b-6aaf-423e-b7ed-2e64dbdfa535\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"
Nov 02 17:30:10.630476 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.630466    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c89d394-c878-4069-b3a9-cd0abc822fec-root\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.630697 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.630489    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9257d59b-6aaf-423e-b7ed-2e64dbdfa535-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-m89vg\" (UID: \"9257d59b-6aaf-423e-b7ed-2e64dbdfa535\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"
Nov 02 17:30:10.630697 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.630566    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c89d394-c878-4069-b3a9-cd0abc822fec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.630803 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.630688    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c89d394-c878-4069-b3a9-cd0abc822fec-node-exporter-wtmp\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.630803 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.630723    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c89d394-c878-4069-b3a9-cd0abc822fec-sys\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.630803 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.630749    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c89d394-c878-4069-b3a9-cd0abc822fec-node-exporter-textfile\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.630803 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.630776    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7scsp\" (UniqueName: \"kubernetes.io/projected/8c89d394-c878-4069-b3a9-cd0abc822fec-kube-api-access-7scsp\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.631017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.630828    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c89d394-c878-4069-b3a9-cd0abc822fec-node-exporter-accelerators-collector-config\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.631017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.630860    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-txlb6\" (UniqueName: \"kubernetes.io/projected/9257d59b-6aaf-423e-b7ed-2e64dbdfa535-kube-api-access-txlb6\") pod \"openshift-state-metrics-7bd7489957-m89vg\" (UID: \"9257d59b-6aaf-423e-b7ed-2e64dbdfa535\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"
Nov 02 17:30:10.731526 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.731432    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9257d59b-6aaf-423e-b7ed-2e64dbdfa535-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-m89vg\" (UID: \"9257d59b-6aaf-423e-b7ed-2e64dbdfa535\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"
Nov 02 17:30:10.731526 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.731488    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9257d59b-6aaf-423e-b7ed-2e64dbdfa535-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-m89vg\" (UID: \"9257d59b-6aaf-423e-b7ed-2e64dbdfa535\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"
Nov 02 17:30:10.731526 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.731511    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c89d394-c878-4069-b3a9-cd0abc822fec-root\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.731800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.731540    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9257d59b-6aaf-423e-b7ed-2e64dbdfa535-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-m89vg\" (UID: \"9257d59b-6aaf-423e-b7ed-2e64dbdfa535\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"
Nov 02 17:30:10.731800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.731566    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c89d394-c878-4069-b3a9-cd0abc822fec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.731800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.731607    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c89d394-c878-4069-b3a9-cd0abc822fec-node-exporter-wtmp\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.731800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.731622    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"root\" (UniqueName: \"kubernetes.io/host-path/8c89d394-c878-4069-b3a9-cd0abc822fec-root\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.731800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.731633    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c89d394-c878-4069-b3a9-cd0abc822fec-sys\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.731800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.731674    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/8c89d394-c878-4069-b3a9-cd0abc822fec-sys\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.731800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.731694    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c89d394-c878-4069-b3a9-cd0abc822fec-node-exporter-textfile\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.731800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.731725    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7scsp\" (UniqueName: \"kubernetes.io/projected/8c89d394-c878-4069-b3a9-cd0abc822fec-kube-api-access-7scsp\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.731800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.731756    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c89d394-c878-4069-b3a9-cd0abc822fec-node-exporter-accelerators-collector-config\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.731800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.731778    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-txlb6\" (UniqueName: \"kubernetes.io/projected/9257d59b-6aaf-423e-b7ed-2e64dbdfa535-kube-api-access-txlb6\") pod \"openshift-state-metrics-7bd7489957-m89vg\" (UID: \"9257d59b-6aaf-423e-b7ed-2e64dbdfa535\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"
Nov 02 17:30:10.732330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.731839    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c89d394-c878-4069-b3a9-cd0abc822fec-node-exporter-tls\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.732330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.731867    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c89d394-c878-4069-b3a9-cd0abc822fec-metrics-client-ca\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.732474 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.732454    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/8c89d394-c878-4069-b3a9-cd0abc822fec-metrics-client-ca\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.732587 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.732566    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/9257d59b-6aaf-423e-b7ed-2e64dbdfa535-metrics-client-ca\") pod \"openshift-state-metrics-7bd7489957-m89vg\" (UID: \"9257d59b-6aaf-423e-b7ed-2e64dbdfa535\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"
Nov 02 17:30:10.732660 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.732645    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-textfile\" (UniqueName: \"kubernetes.io/empty-dir/8c89d394-c878-4069-b3a9-cd0abc822fec-node-exporter-textfile\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.732730 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.732714    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-wtmp\" (UniqueName: \"kubernetes.io/host-path/8c89d394-c878-4069-b3a9-cd0abc822fec-node-exporter-wtmp\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.733354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.733330    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-accelerators-collector-config\" (UniqueName: \"kubernetes.io/configmap/8c89d394-c878-4069-b3a9-cd0abc822fec-node-exporter-accelerators-collector-config\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.734480 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.734433    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/8c89d394-c878-4069-b3a9-cd0abc822fec-node-exporter-kube-rbac-proxy-config\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.734855 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.734834    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-tls\" (UniqueName: \"kubernetes.io/secret/9257d59b-6aaf-423e-b7ed-2e64dbdfa535-openshift-state-metrics-tls\") pod \"openshift-state-metrics-7bd7489957-m89vg\" (UID: \"9257d59b-6aaf-423e-b7ed-2e64dbdfa535\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"
Nov 02 17:30:10.735638 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.735616    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"node-exporter-tls\" (UniqueName: \"kubernetes.io/secret/8c89d394-c878-4069-b3a9-cd0abc822fec-node-exporter-tls\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.741258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.741233    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-state-metrics-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/9257d59b-6aaf-423e-b7ed-2e64dbdfa535-openshift-state-metrics-kube-rbac-proxy-config\") pod \"openshift-state-metrics-7bd7489957-m89vg\" (UID: \"9257d59b-6aaf-423e-b7ed-2e64dbdfa535\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"
Nov 02 17:30:10.744266 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.744242    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:10.757726 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.757706    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7scsp\" (UniqueName: \"kubernetes.io/projected/8c89d394-c878-4069-b3a9-cd0abc822fec-kube-api-access-7scsp\") pod \"node-exporter-r45c5\" (UID: \"8c89d394-c878-4069-b3a9-cd0abc822fec\") " pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.757915 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.757900    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-txlb6\" (UniqueName: \"kubernetes.io/projected/9257d59b-6aaf-423e-b7ed-2e64dbdfa535-kube-api-access-txlb6\") pod \"openshift-state-metrics-7bd7489957-m89vg\" (UID: \"9257d59b-6aaf-423e-b7ed-2e64dbdfa535\") " pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"
Nov 02 17:30:10.772658 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.772636    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"
Nov 02 17:30:10.844271 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.844247    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/node-exporter-r45c5"
Nov 02 17:30:10.858926 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:30:10.858895    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c89d394_c878_4069_b3a9_cd0abc822fec.slice/crio-dbb3b40a54815625e81fdf46da2189af5dfd9b82470f876198269dad5c352695 WatchSource:0}: Error finding container dbb3b40a54815625e81fdf46da2189af5dfd9b82470f876198269dad5c352695: Status 404 returned error can't find the container with id dbb3b40a54815625e81fdf46da2189af5dfd9b82470f876198269dad5c352695
Nov 02 17:30:10.922647 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:10.922605    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:11.104042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:11.103970    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg"]
Nov 02 17:30:11.107245 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:30:11.107206    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9257d59b_6aaf_423e_b7ed_2e64dbdfa535.slice/crio-897ff964df57c023963107ebe137b2d04e0898307306eac8bedd839b52c5c05f WatchSource:0}: Error finding container 897ff964df57c023963107ebe137b2d04e0898307306eac8bedd839b52c5c05f: Status 404 returned error can't find the container with id 897ff964df57c023963107ebe137b2d04e0898307306eac8bedd839b52c5c05f
Nov 02 17:30:11.153521 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:11.153488    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:11.360027 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:11.359931    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:11.551237 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:11.551187    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:11.752394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:11.752360    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:11.755325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:11.755295    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r45c5" event={"ID":"8c89d394-c878-4069-b3a9-cd0abc822fec","Type":"ContainerStarted","Data":"dbb3b40a54815625e81fdf46da2189af5dfd9b82470f876198269dad5c352695"}
Nov 02 17:30:11.758676 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:11.758569    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg" event={"ID":"9257d59b-6aaf-423e-b7ed-2e64dbdfa535","Type":"ContainerStarted","Data":"95aa25a676dab3d451129d2be786caa547466921db7bf846ce9857e6198b2d79"}
Nov 02 17:30:11.758676 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:11.758625    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg" event={"ID":"9257d59b-6aaf-423e-b7ed-2e64dbdfa535","Type":"ContainerStarted","Data":"09c9815856de59207528c8a5d6c6b2a848211a1a3e0018714b6858a68aad41a2"}
Nov 02 17:30:11.758676 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:11.758648    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg" event={"ID":"9257d59b-6aaf-423e-b7ed-2e64dbdfa535","Type":"ContainerStarted","Data":"897ff964df57c023963107ebe137b2d04e0898307306eac8bedd839b52c5c05f"}
Nov 02 17:30:11.933626 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:11.933591    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:12.341343 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.341300    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:12.632402 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.632358    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:12.681927 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.681894    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/thanos-querier-667db95b96-kmwrg"]
Nov 02 17:30:12.688122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.688096    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.723661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.723631    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 17:30:12.724048 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.723924    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 17:30:12.724048 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.723945    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 17:30:12.724048 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.723967    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 17:30:12.724048 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.723971    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-j2fn5\""
Nov 02 17:30:12.724048 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.724020    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5oaelllrcvii0\""
Nov 02 17:30:12.724376 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.724067    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 17:30:12.724591 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.724573    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-667db95b96-kmwrg"]
Nov 02 17:30:12.763914 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.763771    4456 generic.go:356] "Generic (PLEG): container finished" podID="8c89d394-c878-4069-b3a9-cd0abc822fec" containerID="0f1ffa5562e015e8cc5cf6fab3f3b50455a44c8463f68ad6170b52e9a79961a0" exitCode=0
Nov 02 17:30:12.764067 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.763924    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r45c5" event={"ID":"8c89d394-c878-4069-b3a9-cd0abc822fec","Type":"ContainerDied","Data":"0f1ffa5562e015e8cc5cf6fab3f3b50455a44c8463f68ad6170b52e9a79961a0"}
Nov 02 17:30:12.766751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.766685    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg" event={"ID":"9257d59b-6aaf-423e-b7ed-2e64dbdfa535","Type":"ContainerStarted","Data":"cd2e29ab691b451c3990577f92dfc15c008b79756daee1c88781976add8bf024"}
Nov 02 17:30:12.850613 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.850581    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.850735 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.850634    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb44e7b7-e0cb-4270-b14c-9c2397f37269-metrics-client-ca\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.850735 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.850666    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.850735 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.850694    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-grpc-tls\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.850735 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.850720    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tpfs8\" (UniqueName: \"kubernetes.io/projected/fb44e7b7-e0cb-4270-b14c-9c2397f37269-kube-api-access-tpfs8\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.850987 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.850753    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-thanos-querier-tls\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.850987 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.850784    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.850987 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.850835    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.850987 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.850860    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb44e7b7-e0cb-4270-b14c-9c2397f37269-tmp\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.871268 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.871224    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/openshift-state-metrics-7bd7489957-m89vg" podStartSLOduration=1.9454085129999998 podStartE2EDuration="2.871210865s" totalImagesPullingTime="925.802352ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:12.870702451 +0000 UTC m=+29.361765612" watchObservedRunningTime="2026-11-02 17:30:12.871210865 +0000 UTC m=+29.362274022"
Nov 02 17:30:12.951914 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.951868    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.952088 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.952053    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb44e7b7-e0cb-4270-b14c-9c2397f37269-metrics-client-ca\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.952088 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.952085    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.952191 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.952109    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-grpc-tls\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.952191 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.952135    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tpfs8\" (UniqueName: \"kubernetes.io/projected/fb44e7b7-e0cb-4270-b14c-9c2397f37269-kube-api-access-tpfs8\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.952191 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.952185    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-thanos-querier-tls\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.952347 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.952233    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.952347 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.952298    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.952347 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.952339    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb44e7b7-e0cb-4270-b14c-9c2397f37269-tmp\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.952952 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.952922    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/fb44e7b7-e0cb-4270-b14c-9c2397f37269-metrics-client-ca\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.953345 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.953186    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/fb44e7b7-e0cb-4270-b14c-9c2397f37269-tmp\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.955163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.955128    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-metrics\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-thanos-querier-kube-rbac-proxy-metrics\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.955426 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.955404    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-rules\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-thanos-querier-kube-rbac-proxy-rules\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.955657 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.955636    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-thanos-querier-kube-rbac-proxy-web\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.955906 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.955881    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-grpc-tls\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.956411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.956390    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-tls\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-thanos-querier-tls\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.956894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.956853    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-thanos-querier-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/fb44e7b7-e0cb-4270-b14c-9c2397f37269-secret-thanos-querier-kube-rbac-proxy\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:12.977920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:12.977900    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tpfs8\" (UniqueName: \"kubernetes.io/projected/fb44e7b7-e0cb-4270-b14c-9c2397f37269-kube-api-access-tpfs8\") pod \"thanos-querier-667db95b96-kmwrg\" (UID: \"fb44e7b7-e0cb-4270-b14c-9c2397f37269\") " pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:13.017303 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:13.017275    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:13.410860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:13.410805    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/thanos-querier-667db95b96-kmwrg"]
Nov 02 17:30:13.414713 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:30:13.414689    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb44e7b7_e0cb_4270_b14c_9c2397f37269.slice/crio-eb21c694f74029685f9c73d5bd93848bbc2a94caed9fab1d8be4be7f0279a196 WatchSource:0}: Error finding container eb21c694f74029685f9c73d5bd93848bbc2a94caed9fab1d8be4be7f0279a196: Status 404 returned error can't find the container with id eb21c694f74029685f9c73d5bd93848bbc2a94caed9fab1d8be4be7f0279a196
Nov 02 17:30:13.733062 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:13.732970    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:13.771454 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:13.771415    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" event={"ID":"fb44e7b7-e0cb-4270-b14c-9c2397f37269","Type":"ContainerStarted","Data":"eb21c694f74029685f9c73d5bd93848bbc2a94caed9fab1d8be4be7f0279a196"}
Nov 02 17:30:13.774040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:13.773971    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r45c5" event={"ID":"8c89d394-c878-4069-b3a9-cd0abc822fec","Type":"ContainerStarted","Data":"bc4e861f904141308fe5466e3ad28c19ce34724283a4ae3ac27ea89c3cd8afdb"}
Nov 02 17:30:13.774040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:13.774027    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/node-exporter-r45c5" event={"ID":"8c89d394-c878-4069-b3a9-cd0abc822fec","Type":"ContainerStarted","Data":"6c8b037be75ceb4801b5ec6bf216bfbf00a5736b810d128540c838c29b1e7570"}
Nov 02 17:30:13.829313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:13.829260    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/node-exporter-r45c5" podStartSLOduration=2.799749828 podStartE2EDuration="3.829246341s" totalImagesPullingTime="941.707622ms" totalInitContainerRuntime="87.788891ms" isStatefulPod=true podCreationTimestamp="2026-11-02 17:30:10 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:13.828947749 +0000 UTC m=+30.320010912" watchObservedRunningTime="2026-11-02 17:30:13.829246341 +0000 UTC m=+30.320309503"
Nov 02 17:30:13.890181 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:13.890150    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:14.040580 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:14.040460    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:15.031274 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.031217    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:15.143591 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.142416    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/metrics-server-7d987f75-ntp98"]
Nov 02 17:30:15.159643 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.159604    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.169429 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.169408    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/18d2ae0a-bf65-4aee-8730-093e00b54a24-client-ca-bundle\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.169551 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.169459    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18d2ae0a-bf65-4aee-8730-093e00b54a24-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.169619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.169568    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/18d2ae0a-bf65-4aee-8730-093e00b54a24-secret-metrics-server-tls\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.169619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.169606    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/18d2ae0a-bf65-4aee-8730-093e00b54a24-audit-log\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.169728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.169645    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/18d2ae0a-bf65-4aee-8730-093e00b54a24-secret-metrics-server-client-certs\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.169728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.169676    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/18d2ae0a-bf65-4aee-8730-093e00b54a24-metrics-server-audit-profiles\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.169728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.169703    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g4wkw\" (UniqueName: \"kubernetes.io/projected/18d2ae0a-bf65-4aee-8730-093e00b54a24-kube-api-access-g4wkw\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.173009 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.172987    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d987f75-ntp98"]
Nov 02 17:30:15.179084 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.179058    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 17:30:15.179216 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.179196    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 17:30:15.206986 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.206959    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qpmjh\""
Nov 02 17:30:15.207151 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.206991    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 17:30:15.207151 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.207065    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 17:30:15.207355 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.207315    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8hdubeblopv5q\""
Nov 02 17:30:15.216063 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.216037    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:15.269989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.269949    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18d2ae0a-bf65-4aee-8730-093e00b54a24-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.270170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.270005    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/18d2ae0a-bf65-4aee-8730-093e00b54a24-secret-metrics-server-tls\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.270170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.270030    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/18d2ae0a-bf65-4aee-8730-093e00b54a24-audit-log\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.270170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.270063    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/18d2ae0a-bf65-4aee-8730-093e00b54a24-secret-metrics-server-client-certs\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.270170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.270085    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/18d2ae0a-bf65-4aee-8730-093e00b54a24-metrics-server-audit-profiles\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.270170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.270109    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g4wkw\" (UniqueName: \"kubernetes.io/projected/18d2ae0a-bf65-4aee-8730-093e00b54a24-kube-api-access-g4wkw\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.270170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.270142    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/18d2ae0a-bf65-4aee-8730-093e00b54a24-client-ca-bundle\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.270555 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.270528    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"audit-log\" (UniqueName: \"kubernetes.io/empty-dir/18d2ae0a-bf65-4aee-8730-093e00b54a24-audit-log\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.270753 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.270721    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/18d2ae0a-bf65-4aee-8730-093e00b54a24-configmap-kubelet-serving-ca-bundle\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.271346 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.271315    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-server-audit-profiles\" (UniqueName: \"kubernetes.io/configmap/18d2ae0a-bf65-4aee-8730-093e00b54a24-metrics-server-audit-profiles\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.273282 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.273257    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"client-ca-bundle\" (UniqueName: \"kubernetes.io/secret/18d2ae0a-bf65-4aee-8730-093e00b54a24-client-ca-bundle\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.273406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.273304    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-client-certs\" (UniqueName: \"kubernetes.io/secret/18d2ae0a-bf65-4aee-8730-093e00b54a24-secret-metrics-server-client-certs\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.273681 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.273661    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-server-tls\" (UniqueName: \"kubernetes.io/secret/18d2ae0a-bf65-4aee-8730-093e00b54a24-secret-metrics-server-tls\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.295929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.295843    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g4wkw\" (UniqueName: \"kubernetes.io/projected/18d2ae0a-bf65-4aee-8730-093e00b54a24-kube-api-access-g4wkw\") pod \"metrics-server-7d987f75-ntp98\" (UID: \"18d2ae0a-bf65-4aee-8730-093e00b54a24\") " pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.488434 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.488396    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:15.831638 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.831555    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"]
Nov 02 17:30:15.844373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.844318    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/metrics-server-7d987f75-ntp98"]
Nov 02 17:30:15.844530 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.844464    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.864718 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.864695    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qn5gh\""
Nov 02 17:30:15.864827 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.864695    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 02 17:30:15.865078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.865064    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 02 17:30:15.874059 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.874028    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e065a5c3-95fd-4ac8-a287-8e1d20219201-telemeter-client-tls\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.874173 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.874071    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e065a5c3-95fd-4ac8-a287-8e1d20219201-federate-client-tls\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.874173 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.874095    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e065a5c3-95fd-4ac8-a287-8e1d20219201-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.874173 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.874149    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7rpln\" (UniqueName: \"kubernetes.io/projected/e065a5c3-95fd-4ac8-a287-8e1d20219201-kube-api-access-7rpln\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.874330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.874208    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e065a5c3-95fd-4ac8-a287-8e1d20219201-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.874330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.874261    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e065a5c3-95fd-4ac8-a287-8e1d20219201-serving-certs-ca-bundle\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.874330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.874294    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e065a5c3-95fd-4ac8-a287-8e1d20219201-secret-telemeter-client\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.874483 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.874370    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e065a5c3-95fd-4ac8-a287-8e1d20219201-metrics-client-ca\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.890390 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.890365    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 02 17:30:15.890390 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.890373    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 02 17:30:15.890589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.890518    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 02 17:30:15.893851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.893807    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"]
Nov 02 17:30:15.952564 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:30:15.952526    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18d2ae0a_bf65_4aee_8730_093e00b54a24.slice/crio-6512ba51a4b4b808f376f04471d02b6e235e6c45fd1aaf60eaf004ac8c981394 WatchSource:0}: Error finding container 6512ba51a4b4b808f376f04471d02b6e235e6c45fd1aaf60eaf004ac8c981394: Status 404 returned error can't find the container with id 6512ba51a4b4b808f376f04471d02b6e235e6c45fd1aaf60eaf004ac8c981394
Nov 02 17:30:15.965005 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.964972    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 02 17:30:15.975368 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.975346    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e065a5c3-95fd-4ac8-a287-8e1d20219201-serving-certs-ca-bundle\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.975485 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.975377    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e065a5c3-95fd-4ac8-a287-8e1d20219201-secret-telemeter-client\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.975485 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.975404    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e065a5c3-95fd-4ac8-a287-8e1d20219201-metrics-client-ca\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.975485 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.975423    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e065a5c3-95fd-4ac8-a287-8e1d20219201-telemeter-client-tls\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.975485 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.975461    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e065a5c3-95fd-4ac8-a287-8e1d20219201-federate-client-tls\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.975664 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.975485    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e065a5c3-95fd-4ac8-a287-8e1d20219201-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.975664 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.975514    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7rpln\" (UniqueName: \"kubernetes.io/projected/e065a5c3-95fd-4ac8-a287-8e1d20219201-kube-api-access-7rpln\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.975664 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.975549    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e065a5c3-95fd-4ac8-a287-8e1d20219201-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.976209 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.976185    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e065a5c3-95fd-4ac8-a287-8e1d20219201-serving-certs-ca-bundle\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.976329 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.976184    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/e065a5c3-95fd-4ac8-a287-8e1d20219201-metrics-client-ca\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.976597 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.976571    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/e065a5c3-95fd-4ac8-a287-8e1d20219201-telemeter-trusted-ca-bundle\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.978772 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.978748    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"telemeter-client-tls\" (UniqueName: \"kubernetes.io/secret/e065a5c3-95fd-4ac8-a287-8e1d20219201-telemeter-client-tls\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.978961 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.978937    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client\" (UniqueName: \"kubernetes.io/secret/e065a5c3-95fd-4ac8-a287-8e1d20219201-secret-telemeter-client\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.978961 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.978952    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-telemeter-client-kube-rbac-proxy-config\" (UniqueName: \"kubernetes.io/secret/e065a5c3-95fd-4ac8-a287-8e1d20219201-secret-telemeter-client-kube-rbac-proxy-config\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:15.979094 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:15.979011    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"federate-client-tls\" (UniqueName: \"kubernetes.io/secret/e065a5c3-95fd-4ac8-a287-8e1d20219201-federate-client-tls\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:16.006231 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:16.006211    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7rpln\" (UniqueName: \"kubernetes.io/projected/e065a5c3-95fd-4ac8-a287-8e1d20219201-kube-api-access-7rpln\") pod \"telemeter-client-79c994cf5c-ckj5h\" (UID: \"e065a5c3-95fd-4ac8-a287-8e1d20219201\") " pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:16.165127 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:30:16.165098    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:30:16.220481 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:16.220459    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"
Nov 02 17:30:16.587419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:16.587382    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/telemeter-client-79c994cf5c-ckj5h"]
Nov 02 17:30:16.786353 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:16.786319    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h" event={"ID":"e065a5c3-95fd-4ac8-a287-8e1d20219201","Type":"ContainerStarted","Data":"ba8c31a3e996951571f4060d5c2f867c1003025c506cedfbf9b1d88c603a3e76"}
Nov 02 17:30:16.790510 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:16.790473    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d987f75-ntp98" event={"ID":"18d2ae0a-bf65-4aee-8730-093e00b54a24","Type":"ContainerStarted","Data":"6512ba51a4b4b808f376f04471d02b6e235e6c45fd1aaf60eaf004ac8c981394"}
Nov 02 17:30:16.793412 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:16.793384    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" event={"ID":"fb44e7b7-e0cb-4270-b14c-9c2397f37269","Type":"ContainerStarted","Data":"fb7d8fd5c350ff0b6042c18ea0042c92b0b48df0ad64447b907945240a8e9574"}
Nov 02 17:30:16.793536 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:16.793456    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" event={"ID":"fb44e7b7-e0cb-4270-b14c-9c2397f37269","Type":"ContainerStarted","Data":"61f0d172af153366e5fe0177879402cf8b046838c4d8800dd144c8d0095dd565"}
Nov 02 17:30:16.793536 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:16.793488    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" event={"ID":"fb44e7b7-e0cb-4270-b14c-9c2397f37269","Type":"ContainerStarted","Data":"dd72f26848abbdc659729bdded9fd6b07c72255a76ba4db020565307035358aa"}
Nov 02 17:30:17.084071 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.083993    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:17.280145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.280101    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:17.642308 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.642277    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 17:30:17.657884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.656224    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.691911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.691886    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8dlk7\" (UniqueName: \"kubernetes.io/projected/b04df333-4a88-41aa-9e81-41a5cb5c0c64-kube-api-access-8dlk7\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.691927    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.691958    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.691987    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692032    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692061    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-config\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692090    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692150    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-web-config\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692375 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692179    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b04df333-4a88-41aa-9e81-41a5cb5c0c64-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692375 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692205    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b04df333-4a88-41aa-9e81-41a5cb5c0c64-config-out\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692375 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692227    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692375 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692282    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692375 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692312    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692375 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692339    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692375 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692369    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692688 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692395    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692688 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692437    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692688 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692463    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692688 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692498    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.692688 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.692526    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.707042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.707001    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 17:30:17.707449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.707419    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 17:30:17.710287 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.710267    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 17:30:17.710457 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.710438    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mw4s9\""
Nov 02 17:30:17.711621 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.711605    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5mdnhf4neu495\""
Nov 02 17:30:17.748856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.748832    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 17:30:17.760803 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.760690    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 17:30:17.760803 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.760784    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 17:30:17.761266 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.761238    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 17:30:17.763399 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.762875    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 17:30:17.763399 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.762918    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 17:30:17.763399 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.762957    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 17:30:17.764612 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.764539    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 17:30:17.773569 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.773485    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 17:30:17.774255 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.774229    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 17:30:17.784059 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.784030    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:17.789709 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.789431    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 17:30:17.789709 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.789679    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793177    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793225    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793263    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-config\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793291    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793316    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-web-config\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793346    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b04df333-4a88-41aa-9e81-41a5cb5c0c64-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793369    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b04df333-4a88-41aa-9e81-41a5cb5c0c64-config-out\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793392    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793436    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793464    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793491    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793533    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793558    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793599    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793626    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793667    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793714    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.793797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793737    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8dlk7\" (UniqueName: \"kubernetes.io/projected/b04df333-4a88-41aa-9e81-41a5cb5c0c64-kube-api-access-8dlk7\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.794762 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.793891    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.796994 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.795620    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.796994 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.796625    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.796994 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.796783    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.797653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.797219    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.797653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.797272    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.797653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.797581    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.800757 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.800730    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.802347 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.802295    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.804187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.802715    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-config\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.804187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.803359    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.804187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.803561    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b04df333-4a88-41aa-9e81-41a5cb5c0c64-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.804187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.804125    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.804414 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.804195    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.805577 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.804624    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.805577 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.804669    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-web-config\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.805577 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.804872    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b04df333-4a88-41aa-9e81-41a5cb5c0c64-config-out\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.805577 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.805235    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.805577 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.805413    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.808420 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.807998    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.811267 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.811183    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:17.816194 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.816169    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" event={"ID":"fb44e7b7-e0cb-4270-b14c-9c2397f37269","Type":"ContainerStarted","Data":"0f57da1c99b1751c6e320dcecf275e65ef5a73fa649f70b7f3519f90a4a45a30"}
Nov 02 17:30:17.816316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.816299    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" event={"ID":"fb44e7b7-e0cb-4270-b14c-9c2397f37269","Type":"ContainerStarted","Data":"6f71007c295e586385ed47cbedb1f508d12587db8b68c90755b17765b757ea51"}
Nov 02 17:30:17.840926 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:17.840902    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8dlk7\" (UniqueName: \"kubernetes.io/projected/b04df333-4a88-41aa-9e81-41a5cb5c0c64-kube-api-access-8dlk7\") pod \"prometheus-k8s-0\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:18.080808 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:18.080494    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:18.094049 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:18.093707    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-network-diagnostics/network-check-target-nttxj"
Nov 02 17:30:18.459741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:18.459703    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 17:30:18.476745 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:30:18.476703    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb04df333_4a88_41aa_9e81_41a5cb5c0c64.slice/crio-9c96d6d7de612c29cfb5877b5dae6bc5d784114669e2599d49e5e2e15d467d21 WatchSource:0}: Error finding container 9c96d6d7de612c29cfb5877b5dae6bc5d784114669e2599d49e5e2e15d467d21: Status 404 returned error can't find the container with id 9c96d6d7de612c29cfb5877b5dae6bc5d784114669e2599d49e5e2e15d467d21
Nov 02 17:30:18.820848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:18.820755    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d987f75-ntp98" event={"ID":"18d2ae0a-bf65-4aee-8730-093e00b54a24","Type":"ContainerStarted","Data":"c426b7d4950263c913e37548ad7992f0212cc9969a4598d6b4fc3ff560c31e70"}
Nov 02 17:30:18.822102 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:18.822073    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerStarted","Data":"9c96d6d7de612c29cfb5877b5dae6bc5d784114669e2599d49e5e2e15d467d21"}
Nov 02 17:30:18.825013 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:18.824994    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" event={"ID":"fb44e7b7-e0cb-4270-b14c-9c2397f37269","Type":"ContainerStarted","Data":"4128cd8c2d39877969f13c516475d9c81d5ed650a72453945483c1ab1b0fb71f"}
Nov 02 17:30:18.825201 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:18.825186    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:18.869038 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:18.868986    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/metrics-server-7d987f75-ntp98" podStartSLOduration=1.89102508 podStartE2EDuration="3.868971757s" totalImagesPullingTime="1.977946677s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:15 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:18.868908428 +0000 UTC m=+35.359971591" watchObservedRunningTime="2026-11-02 17:30:18.868971757 +0000 UTC m=+35.360034920"
Nov 02 17:30:18.922722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:18.922656    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podStartSLOduration=3.077373235 podStartE2EDuration="6.922644612s" totalImagesPullingTime="3.845271377s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:12 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:18.921401159 +0000 UTC m=+35.412464351" watchObservedRunningTime="2026-11-02 17:30:18.922644612 +0000 UTC m=+35.413707797"
Nov 02 17:30:19.839317 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:19.839228    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h" event={"ID":"e065a5c3-95fd-4ac8-a287-8e1d20219201","Type":"ContainerStarted","Data":"e1dea9c94e305f5df73b99150093ac5f32d213bfa5441e23e2be0a9c84fee85a"}
Nov 02 17:30:19.843150 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:19.841983    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-pgmzg" event={"ID":"6e78d6c7-c325-48f2-af90-804dade84d8f","Type":"ContainerStarted","Data":"4cee89e694b5799548d6085cbcaa2612948ac905b236f27ac06a462827cad81c"}
Nov 02 17:30:19.843150 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:19.842965    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-pgmzg"
Nov 02 17:30:19.846583 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:19.846537    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerStarted","Data":"84d6d528290bb8ffad2dea582f489608ac9af68ac57831c597705aa856a1a968"}
Nov 02 17:30:19.857934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:19.857913    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-console/downloads-6dd789c8b7-pgmzg"
Nov 02 17:30:19.888185 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:19.888129    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podStartSLOduration=1.949170963 podStartE2EDuration="18.888111663s" totalImagesPullingTime="16.9389407s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:19.886535534 +0000 UTC m=+36.377598731" watchObservedRunningTime="2026-11-02 17:30:19.888111663 +0000 UTC m=+36.379174824"
Nov 02 17:30:20.853359 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:20.853313    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h" event={"ID":"e065a5c3-95fd-4ac8-a287-8e1d20219201","Type":"ContainerStarted","Data":"0f7c317c8e613f88bf53bc5648d2523f0ea65088306722ee7e82bc70c78cd7f5"}
Nov 02 17:30:20.854025 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:20.853420    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h" event={"ID":"e065a5c3-95fd-4ac8-a287-8e1d20219201","Type":"ContainerStarted","Data":"8c097a036400f82a13f8dfa5e7d8199c43e3833d7c31df049b9c9988ae96de13"}
Nov 02 17:30:20.855529 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:20.855495    4456 generic.go:356] "Generic (PLEG): container finished" podID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerID="84d6d528290bb8ffad2dea582f489608ac9af68ac57831c597705aa856a1a968" exitCode=0
Nov 02 17:30:20.855668 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:20.855576    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerDied","Data":"84d6d528290bb8ffad2dea582f489608ac9af68ac57831c597705aa856a1a968"}
Nov 02 17:30:20.939258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:20.939195    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/telemeter-client-79c994cf5c-ckj5h" podStartSLOduration=2.880671791 podStartE2EDuration="5.939177163s" totalImagesPullingTime="3.058505372s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:15 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:20.926227868 +0000 UTC m=+37.417291040" watchObservedRunningTime="2026-11-02 17:30:20.939177163 +0000 UTC m=+37.430240324"
Nov 02 17:30:22.241331 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:22.241218    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:22.390728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:22.390623    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:24.057482 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:24.057384    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:24.221331 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:24.221296    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:24.735014 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:24.734987    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 17:30:24.870299 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:24.869082    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 17:30:24.884902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:24.884858    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerStarted","Data":"bda787c756a6989d5e69f71220f0be9e4ec7e0053356c0723c45d4bef5f3d993"}
Nov 02 17:30:24.885022 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:24.884967    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerStarted","Data":"1669caf0461a128092420698545192319fa52517d9d2db4bab7d45cc98818b2c"}
Nov 02 17:30:25.063835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:25.062061    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:25.224046 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:25.224010    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:25.893938 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:25.893897    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerStarted","Data":"1996c624dc427c0303019be617c68af03fc3980f00417ae74f6692cd00b6b0f7"}
Nov 02 17:30:25.894142 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:25.893976    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerStarted","Data":"ffd337cae5bbb75eeef40c2ef1c0b53d1904113995f55fbed973c2aa33897bdf"}
Nov 02 17:30:25.894142 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:25.894017    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerStarted","Data":"a62210baa064c1cf8fc82c362402036ed04c37780706eca9b09081ddefd4467b"}
Nov 02 17:30:25.894142 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:25.894054    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerStarted","Data":"60ef41d5a66269d85bab12adf22f78433318601954a8d1dd652eea3e6c27389b"}
Nov 02 17:30:25.991765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:25.991713    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=3.968829885 podStartE2EDuration="8.991700358s" totalImagesPullingTime="4.987281536s" totalInitContainerRuntime="35.588937ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:30:17 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:30:25.991325846 +0000 UTC m=+42.482389008" watchObservedRunningTime="2026-11-02 17:30:25.991700358 +0000 UTC m=+42.482763518"
Nov 02 17:30:27.655475 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:27.655432    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:27.808096 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:27.808052    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:28.081137 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:28.081098    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:29.858355 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:29.858322    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:30.039836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:30.039781    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:30.264197 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:30.264156    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:30.450501 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:30.450469    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:32.072355 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:32.072316    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:32.337714 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:32.337631    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:34.510127 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:34.510083    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:34.722506 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:34.722475    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:34.908433 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:34.908349    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:35.078100 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:35.078070    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:35.489215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:35.489181    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:35.489512 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:35.489250    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:36.659859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:36.659827    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:36.817088 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:36.817060    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:44.894877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:44.894843    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:45.036349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:45.036314    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:46.163855 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:30:46.163805    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:30:50.387104 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:50.387062    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:50.436764 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:50.436735    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:51.004391 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:51.004363    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:30:55.493703 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:55.493675    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:55.497388 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:55.497369    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 17:30:57.282914 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:57.282872    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:57.443421 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:57.443387    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:59.819329 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:59.819294    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:30:59.971672 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:30:59.971640    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:01.102113 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:01.102070    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:04.038372 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:04.038334    4456 generic.go:356] "Generic (PLEG): container finished" podID="83455bdf-65dc-4f66-b943-55835e92b360" containerID="28db33cefa834b81d5b885b6a0ec776bb6e5248b0d66361db777a80d79851e25" exitCode=0
Nov 02 17:31:04.038730 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:04.038399    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wphg6-wf57n" event={"ID":"83455bdf-65dc-4f66-b943-55835e92b360","Type":"ContainerDied","Data":"28db33cefa834b81d5b885b6a0ec776bb6e5248b0d66361db777a80d79851e25"}
Nov 02 17:31:04.452348 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:04.452310    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:04.630517 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:04.630470    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:04.776563 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:04.776491    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:05.199898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:05.199868    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:31:05.261044 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:05.261015    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-service-ca-bundle\") pod \"83455bdf-65dc-4f66-b943-55835e92b360\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") "
Nov 02 17:31:05.261219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:05.261079    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-trusted-ca-bundle\") pod \"83455bdf-65dc-4f66-b943-55835e92b360\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") "
Nov 02 17:31:05.261219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:05.261107    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/83455bdf-65dc-4f66-b943-55835e92b360-kube-api-access-l4kfc\" (UniqueName: \"kubernetes.io/projected/83455bdf-65dc-4f66-b943-55835e92b360-kube-api-access-l4kfc\") pod \"83455bdf-65dc-4f66-b943-55835e92b360\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") "
Nov 02 17:31:05.261219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:05.261201    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/83455bdf-65dc-4f66-b943-55835e92b360-archives-path\" (UniqueName: \"kubernetes.io/empty-dir/83455bdf-65dc-4f66-b943-55835e92b360-archives-path\") pod \"83455bdf-65dc-4f66-b943-55835e92b360\" (UID: \"83455bdf-65dc-4f66-b943-55835e92b360\") "
Nov 02 17:31:05.261486 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:05.261464    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-service-ca-bundle" pod "83455bdf-65dc-4f66-b943-55835e92b360" (UID: "83455bdf-65dc-4f66-b943-55835e92b360"). InnerVolumeSpecName "service-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:31:05.261549 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:05.261482    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-trusted-ca-bundle" pod "83455bdf-65dc-4f66-b943-55835e92b360" (UID: "83455bdf-65dc-4f66-b943-55835e92b360"). InnerVolumeSpecName "trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:31:05.261601 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:05.261573    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/83455bdf-65dc-4f66-b943-55835e92b360-archives-path" pod "83455bdf-65dc-4f66-b943-55835e92b360" (UID: "83455bdf-65dc-4f66-b943-55835e92b360"). InnerVolumeSpecName "archives-path". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:31:05.263612 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:05.263592    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/83455bdf-65dc-4f66-b943-55835e92b360-kube-api-access-l4kfc" pod "83455bdf-65dc-4f66-b943-55835e92b360" (UID: "83455bdf-65dc-4f66-b943-55835e92b360"). InnerVolumeSpecName "kube-api-access-l4kfc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:31:05.361856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:05.361762    4456 reconciler_common.go:299] "Volume detached for volume \"service-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-service-ca-bundle\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:05.361856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:05.361790    4456 reconciler_common.go:299] "Volume detached for volume \"trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83455bdf-65dc-4f66-b943-55835e92b360-trusted-ca-bundle\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:05.361856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:05.361799    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4kfc\" (UniqueName: \"kubernetes.io/projected/83455bdf-65dc-4f66-b943-55835e92b360-kube-api-access-l4kfc\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:05.361856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:05.361810    4456 reconciler_common.go:299] "Volume detached for volume \"archives-path\" (UniqueName: \"kubernetes.io/empty-dir/83455bdf-65dc-4f66-b943-55835e92b360-archives-path\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:06.046678 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:06.046650    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-insights/periodic-gathering-wphg6-wf57n"
Nov 02 17:31:06.046859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:06.046643    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-insights/periodic-gathering-wphg6-wf57n" event={"ID":"83455bdf-65dc-4f66-b943-55835e92b360","Type":"ContainerDied","Data":"00c2ae5d68441defef474549a48ed21d2249faeefb1b703c42569e621f3a57d2"}
Nov 02 17:31:06.046859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:06.046787    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="00c2ae5d68441defef474549a48ed21d2249faeefb1b703c42569e621f3a57d2"
Nov 02 17:31:06.741979 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:06.741940    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:06.885133 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:06.885102    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:06.957311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:06.957279    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 17:31:06.957799 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:06.957758    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="prometheus" containerID="cri-o://1669caf0461a128092420698545192319fa52517d9d2db4bab7d45cc98818b2c" gracePeriod=600
Nov 02 17:31:06.957968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:06.957799    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="kube-rbac-proxy-thanos" containerID="cri-o://1996c624dc427c0303019be617c68af03fc3980f00417ae74f6692cd00b6b0f7" gracePeriod=600
Nov 02 17:31:06.957968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:06.957763    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="kube-rbac-proxy" containerID="cri-o://ffd337cae5bbb75eeef40c2ef1c0b53d1904113995f55fbed973c2aa33897bdf" gracePeriod=600
Nov 02 17:31:06.957968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:06.957880    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="config-reloader" containerID="cri-o://bda787c756a6989d5e69f71220f0be9e4ec7e0053356c0723c45d4bef5f3d993" gracePeriod=600
Nov 02 17:31:06.958165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:06.957888    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="kube-rbac-proxy-web" containerID="cri-o://a62210baa064c1cf8fc82c362402036ed04c37780706eca9b09081ddefd4467b" gracePeriod=600
Nov 02 17:31:06.958165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:06.957880    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-k8s-0" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="thanos-sidecar" containerID="cri-o://60ef41d5a66269d85bab12adf22f78433318601954a8d1dd652eea3e6c27389b" gracePeriod=600
Nov 02 17:31:08.059193 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:08.059163    4456 generic.go:356] "Generic (PLEG): container finished" podID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerID="a62210baa064c1cf8fc82c362402036ed04c37780706eca9b09081ddefd4467b" exitCode=0
Nov 02 17:31:08.059522 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:08.059196    4456 generic.go:356] "Generic (PLEG): container finished" podID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerID="60ef41d5a66269d85bab12adf22f78433318601954a8d1dd652eea3e6c27389b" exitCode=0
Nov 02 17:31:08.059522 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:08.059206    4456 generic.go:356] "Generic (PLEG): container finished" podID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerID="bda787c756a6989d5e69f71220f0be9e4ec7e0053356c0723c45d4bef5f3d993" exitCode=0
Nov 02 17:31:08.059522 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:08.059230    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerDied","Data":"a62210baa064c1cf8fc82c362402036ed04c37780706eca9b09081ddefd4467b"}
Nov 02 17:31:08.059522 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:08.059306    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerDied","Data":"60ef41d5a66269d85bab12adf22f78433318601954a8d1dd652eea3e6c27389b"}
Nov 02 17:31:08.059522 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:08.059354    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerDied","Data":"bda787c756a6989d5e69f71220f0be9e4ec7e0053356c0723c45d4bef5f3d993"}
Nov 02 17:31:09.066884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:09.066855    4456 generic.go:356] "Generic (PLEG): container finished" podID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerID="1996c624dc427c0303019be617c68af03fc3980f00417ae74f6692cd00b6b0f7" exitCode=0
Nov 02 17:31:09.066884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:09.066880    4456 generic.go:356] "Generic (PLEG): container finished" podID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerID="ffd337cae5bbb75eeef40c2ef1c0b53d1904113995f55fbed973c2aa33897bdf" exitCode=0
Nov 02 17:31:09.067293 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:09.066908    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerDied","Data":"1996c624dc427c0303019be617c68af03fc3980f00417ae74f6692cd00b6b0f7"}
Nov 02 17:31:09.067293 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:09.066959    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerDied","Data":"ffd337cae5bbb75eeef40c2ef1c0b53d1904113995f55fbed973c2aa33897bdf"}
Nov 02 17:31:10.988714 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:31:10.988663    4456 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="1669caf0461a128092420698545192319fa52517d9d2db4bab7d45cc98818b2c" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 02 17:31:10.989113 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:31:10.988734    4456 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="1669caf0461a128092420698545192319fa52517d9d2db4bab7d45cc98818b2c" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 02 17:31:10.989113 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:31:10.988758    4456 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = Canceled desc = context canceled" containerID="1669caf0461a128092420698545192319fa52517d9d2db4bab7d45cc98818b2c" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 02 17:31:10.989113 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:10.988777    4456 prober.go:106] "Probe errored" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="prometheus" probeResult="unknown" err="rpc error: code = Canceled desc = context canceled"
Nov 02 17:31:14.090587 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.090546    4456 generic.go:356] "Generic (PLEG): container finished" podID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerID="1669caf0461a128092420698545192319fa52517d9d2db4bab7d45cc98818b2c" exitCode=0
Nov 02 17:31:14.091003 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.090620    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerDied","Data":"1669caf0461a128092420698545192319fa52517d9d2db4bab7d45cc98818b2c"}
Nov 02 17:31:14.112141 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.112118    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:14.243888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.243860    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-trusted-ca-bundle\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.243888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.243892    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-metrics-client-ca\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.244131 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.243924    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-thanos-prometheus-http-client-file\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.244131 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.243959    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-kube-rbac-proxy\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.244131 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.243987    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.244131 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.244035    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.244131 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.244064    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-2\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.244419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.244387    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-trusted-ca-bundle" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "prometheus-trusted-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:31:14.244509 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.244411    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-2" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "prometheus-k8s-rulefiles-2". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:31:14.244564 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.244514    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-grpc-tls\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.244691 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.244675    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-serving-certs-ca-bundle\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.244768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.244731    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-metrics-client-certs\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.244768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.244739    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-metrics-client-ca" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "configmap-metrics-client-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:31:14.244932 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.244766    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04df333-4a88-41aa-9e81-41a5cb5c0c64-config-out\" (UniqueName: \"kubernetes.io/empty-dir/b04df333-4a88-41aa-9e81-41a5cb5c0c64-config-out\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.244932 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.244800    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04df333-4a88-41aa-9e81-41a5cb5c0c64-tls-assets\" (UniqueName: \"kubernetes.io/projected/b04df333-4a88-41aa-9e81-41a5cb5c0c64-tls-assets\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.244932 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.244853    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-1\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.244932 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.244880    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-db\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.244932 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.244919    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-config\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-config\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.245180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.244946    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-web-config\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-web-config\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.245180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.244975    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-kubelet-serving-ca-bundle\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.245180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.245016    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-tls\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.245180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.245050    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b04df333-4a88-41aa-9e81-41a5cb5c0c64-kube-api-access-8dlk7\" (UniqueName: \"kubernetes.io/projected/b04df333-4a88-41aa-9e81-41a5cb5c0c64-kube-api-access-8dlk7\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.245180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.245086    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-0\") pod \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\" (UID: \"b04df333-4a88-41aa-9e81-41a5cb5c0c64\") "
Nov 02 17:31:14.245411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.245279    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-serving-certs-ca-bundle" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "configmap-serving-certs-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:31:14.245411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.245384    4456 reconciler_common.go:299] "Volume detached for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-trusted-ca-bundle\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.245411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.245405    4456 reconciler_common.go:299] "Volume detached for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-metrics-client-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.245561 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.245421    4456 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-2\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.245561 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.245438    4456 reconciler_common.go:299] "Volume detached for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-serving-certs-ca-bundle\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.247361 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.247315    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-kubelet-serving-ca-bundle" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "configmap-kubelet-serving-ca-bundle". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:31:14.247471 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.247447    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-db" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "prometheus-k8s-db". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:31:14.247530 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.247473    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-kube-rbac-proxy" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "secret-kube-rbac-proxy". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:31:14.247584 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.247540    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-kube-rbac-proxy-web" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "secret-prometheus-k8s-kube-rbac-proxy-web". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:31:14.247920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.247646    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-thanos-prometheus-http-client-file" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "thanos-prometheus-http-client-file". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:31:14.248042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.248009    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-1" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "prometheus-k8s-rulefiles-1". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:31:14.248703 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.248678    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-thanos-sidecar-tls" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "secret-prometheus-k8s-thanos-sidecar-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:31:14.249354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.249329    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-0" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "prometheus-k8s-rulefiles-0". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:31:14.249471 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.249442    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04df333-4a88-41aa-9e81-41a5cb5c0c64-kube-api-access-8dlk7" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "kube-api-access-8dlk7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:31:14.250129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.249970    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/b04df333-4a88-41aa-9e81-41a5cb5c0c64-config-out" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "config-out". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:31:14.250129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.249808    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-tls" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "secret-prometheus-k8s-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:31:14.250129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.250096    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b04df333-4a88-41aa-9e81-41a5cb5c0c64-tls-assets" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "tls-assets". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:31:14.250428 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.250398    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-grpc-tls" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "secret-grpc-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:31:14.250536 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.250423    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-metrics-client-certs" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "secret-metrics-client-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:31:14.251240 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.251214    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-config" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:31:14.262318 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.262296    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-web-config" pod "b04df333-4a88-41aa-9e81-41a5cb5c0c64" (UID: "b04df333-4a88-41aa-9e81-41a5cb5c0c64"). InnerVolumeSpecName "web-config". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:31:14.345983 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.345949    4456 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-1\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.345983 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.345977    4456 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-db\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.345983 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.345987    4456 reconciler_common.go:299] "Volume detached for volume \"config\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-config\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.346169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.345996    4456 reconciler_common.go:299] "Volume detached for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-web-config\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.346169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.346006    4456 reconciler_common.go:299] "Volume detached for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-configmap-kubelet-serving-ca-bundle\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.346169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.346014    4456 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-tls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.346169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.346022    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8dlk7\" (UniqueName: \"kubernetes.io/projected/b04df333-4a88-41aa-9e81-41a5cb5c0c64-kube-api-access-8dlk7\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.346169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.346030    4456 reconciler_common.go:299] "Volume detached for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/b04df333-4a88-41aa-9e81-41a5cb5c0c64-prometheus-k8s-rulefiles-0\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.346169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.346038    4456 reconciler_common.go:299] "Volume detached for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-thanos-prometheus-http-client-file\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.346169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.346046    4456 reconciler_common.go:299] "Volume detached for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-kube-rbac-proxy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.346169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.346056    4456 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-thanos-sidecar-tls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.346169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.346064    4456 reconciler_common.go:299] "Volume detached for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-prometheus-k8s-kube-rbac-proxy-web\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.346169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.346074    4456 reconciler_common.go:299] "Volume detached for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-grpc-tls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.346169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.346081    4456 reconciler_common.go:299] "Volume detached for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/b04df333-4a88-41aa-9e81-41a5cb5c0c64-secret-metrics-client-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.346169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.346090    4456 reconciler_common.go:299] "Volume detached for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/b04df333-4a88-41aa-9e81-41a5cb5c0c64-config-out\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.346169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.346097    4456 reconciler_common.go:299] "Volume detached for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/b04df333-4a88-41aa-9e81-41a5cb5c0c64-tls-assets\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:31:14.834503 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.834457    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:14.984193 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:14.984160    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:15.097802 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.097728    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"b04df333-4a88-41aa-9e81-41a5cb5c0c64","Type":"ContainerDied","Data":"9c96d6d7de612c29cfb5877b5dae6bc5d784114669e2599d49e5e2e15d467d21"}
Nov 02 17:31:15.097802 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.097745    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.097802 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.097803    4456 scope.go:122] "RemoveContainer" containerID="1996c624dc427c0303019be617c68af03fc3980f00417ae74f6692cd00b6b0f7"
Nov 02 17:31:15.105785 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.105763    4456 scope.go:122] "RemoveContainer" containerID="ffd337cae5bbb75eeef40c2ef1c0b53d1904113995f55fbed973c2aa33897bdf"
Nov 02 17:31:15.112766 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.112748    4456 scope.go:122] "RemoveContainer" containerID="a62210baa064c1cf8fc82c362402036ed04c37780706eca9b09081ddefd4467b"
Nov 02 17:31:15.119456 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.119436    4456 scope.go:122] "RemoveContainer" containerID="60ef41d5a66269d85bab12adf22f78433318601954a8d1dd652eea3e6c27389b"
Nov 02 17:31:15.125722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.125704    4456 scope.go:122] "RemoveContainer" containerID="bda787c756a6989d5e69f71220f0be9e4ec7e0053356c0723c45d4bef5f3d993"
Nov 02 17:31:15.132402 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.132386    4456 scope.go:122] "RemoveContainer" containerID="1669caf0461a128092420698545192319fa52517d9d2db4bab7d45cc98818b2c"
Nov 02 17:31:15.139382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.139363    4456 scope.go:122] "RemoveContainer" containerID="84d6d528290bb8ffad2dea582f489608ac9af68ac57831c597705aa856a1a968"
Nov 02 17:31:15.167971 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.167944    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 17:31:15.181060 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.181024    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 17:31:15.210828 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.210780    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 17:31:15.211073 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211061    4456 cpu_manager.go:414] "containerMap: removing container" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="thanos-sidecar"
Nov 02 17:31:15.211114 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211074    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="thanos-sidecar"
Nov 02 17:31:15.211114 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211086    4456 cpu_manager.go:414] "containerMap: removing container" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="prometheus"
Nov 02 17:31:15.211114 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211091    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="prometheus"
Nov 02 17:31:15.211114 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211103    4456 cpu_manager.go:414] "containerMap: removing container" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="config-reloader"
Nov 02 17:31:15.211114 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211108    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="config-reloader"
Nov 02 17:31:15.211114 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211116    4456 cpu_manager.go:414] "containerMap: removing container" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="kube-rbac-proxy-thanos"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211120    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="kube-rbac-proxy-thanos"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211130    4456 cpu_manager.go:414] "containerMap: removing container" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="init-config-reloader"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211135    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="init-config-reloader"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211141    4456 cpu_manager.go:414] "containerMap: removing container" podUID="83455bdf-65dc-4f66-b943-55835e92b360" containerName="insights-gathering"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211145    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="83455bdf-65dc-4f66-b943-55835e92b360" containerName="insights-gathering"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211153    4456 cpu_manager.go:414] "containerMap: removing container" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="kube-rbac-proxy-web"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211158    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="kube-rbac-proxy-web"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211163    4456 cpu_manager.go:414] "containerMap: removing container" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="kube-rbac-proxy"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211168    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="kube-rbac-proxy"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211212    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="kube-rbac-proxy-web"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211218    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="kube-rbac-proxy"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211225    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="thanos-sidecar"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211230    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="prometheus"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211236    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="config-reloader"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211242    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" containerName="kube-rbac-proxy-thanos"
Nov 02 17:31:15.211304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.211247    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="83455bdf-65dc-4f66-b943-55835e92b360" containerName="insights-gathering"
Nov 02 17:31:15.214931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.214913    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.231271 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.231245    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 17:31:15.231452 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.231431    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 17:31:15.231452 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.231442    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 17:31:15.231553 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.231472    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 17:31:15.231778 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.231759    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 17:31:15.231778 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.231766    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 17:31:15.231946 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.231781    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 17:31:15.231946 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.231883    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 17:31:15.231946 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.231902    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 17:31:15.232073 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.231971    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mw4s9\""
Nov 02 17:31:15.232073 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.232006    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 17:31:15.232073 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.232020    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 17:31:15.232073 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.232049    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 17:31:15.245167 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.245002    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5mdnhf4neu495\""
Nov 02 17:31:15.245666 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.245648    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 17:31:15.246755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.246693    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 17:31:15.249057 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.249031    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 17:31:15.354686 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.354597    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.354686 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.354639    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-web-config\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.354895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.354660    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83deda5b-6ef6-4a66-a76c-888303e6e834-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.354895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.354751    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.354895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.354868    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.354988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.354912    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.354988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.354933    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tgbbk\" (UniqueName: \"kubernetes.io/projected/83deda5b-6ef6-4a66-a76c-888303e6e834-kube-api-access-tgbbk\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.354988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.354978    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.355072 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.355030    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.355105 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.355074    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.355105 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.355096    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.355163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.355117    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-config\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.355163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.355134    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83deda5b-6ef6-4a66-a76c-888303e6e834-config-out\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.355241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.355229    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.355275 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.355250    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.355275 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.355269    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.355333 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.355291    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.355387 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.355377    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83deda5b-6ef6-4a66-a76c-888303e6e834-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.355417 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.355396    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.355417 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.355410    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456341    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tgbbk\" (UniqueName: \"kubernetes.io/projected/83deda5b-6ef6-4a66-a76c-888303e6e834-kube-api-access-tgbbk\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456466 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456378    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456466 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456395    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456556 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456534    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456597 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456569    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456647 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456622    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-config\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456700 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456646    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83deda5b-6ef6-4a66-a76c-888303e6e834-config-out\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456700 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456683    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456799 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456709    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456799 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456737    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456799 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456770    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456857    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83deda5b-6ef6-4a66-a76c-888303e6e834-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456883    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456904    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456937    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.456988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.456967    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-web-config\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.457237 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.457002    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83deda5b-6ef6-4a66-a76c-888303e6e834-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.457237 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.457025    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.457237 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.457057    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.457237 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.457100    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.457438 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.457406    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-metrics-client-ca\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-configmap-metrics-client-ca\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.458229 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.458201    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-2\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-prometheus-k8s-rulefiles-2\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.458382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.458203    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-kubelet-serving-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-configmap-kubelet-serving-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.458573 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.458424    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-1\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-prometheus-k8s-rulefiles-1\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.458573 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.458530    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-db\" (UniqueName: \"kubernetes.io/empty-dir/83deda5b-6ef6-4a66-a76c-888303e6e834-prometheus-k8s-db\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.460145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.460118    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-trusted-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-prometheus-trusted-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.460740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.460414    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"configmap-serving-certs-ca-bundle\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-configmap-serving-certs-ca-bundle\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.460740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.460724    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-kube-rbac-proxy-web\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-prometheus-k8s-kube-rbac-proxy-web\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.461627 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.461148    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-tls\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-prometheus-k8s-tls\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.461627 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.461472    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-metrics-client-certs\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-metrics-client-certs\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.461756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.461632    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-kube-rbac-proxy\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-kube-rbac-proxy\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.461854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.461808    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-out\" (UniqueName: \"kubernetes.io/empty-dir/83deda5b-6ef6-4a66-a76c-888303e6e834-config-out\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.462218 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.462174    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-grpc-tls\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-grpc-tls\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.462317 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.462291    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"secret-prometheus-k8s-thanos-sidecar-tls\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-secret-prometheus-k8s-thanos-sidecar-tls\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.462387 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.462328    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"web-config\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-web-config\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.462586 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.462567    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-config\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.463066 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.463052    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-assets\" (UniqueName: \"kubernetes.io/projected/83deda5b-6ef6-4a66-a76c-888303e6e834-tls-assets\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.463346 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.463324    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"prometheus-k8s-rulefiles-0\" (UniqueName: \"kubernetes.io/configmap/83deda5b-6ef6-4a66-a76c-888303e6e834-prometheus-k8s-rulefiles-0\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.464020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.464004    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"thanos-prometheus-http-client-file\" (UniqueName: \"kubernetes.io/secret/83deda5b-6ef6-4a66-a76c-888303e6e834-thanos-prometheus-http-client-file\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.481130 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.481111    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tgbbk\" (UniqueName: \"kubernetes.io/projected/83deda5b-6ef6-4a66-a76c-888303e6e834-kube-api-access-tgbbk\") pod \"prometheus-k8s-0\" (UID: \"83deda5b-6ef6-4a66-a76c-888303e6e834\") " pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.532421 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.532394    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:15.612081 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.611998    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b04df333-4a88-41aa-9e81-41a5cb5c0c64" path="/var/lib/kubelet/pods/b04df333-4a88-41aa-9e81-41a5cb5c0c64/volumes"
Nov 02 17:31:15.880926 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:15.880899    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-monitoring/prometheus-k8s-0"]
Nov 02 17:31:15.883262 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:31:15.883237    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod83deda5b_6ef6_4a66_a76c_888303e6e834.slice/crio-8cc7a09e488bc539f16e64174a11064532b39d8e50b2183b375c7fa660bf984e WatchSource:0}: Error finding container 8cc7a09e488bc539f16e64174a11064532b39d8e50b2183b375c7fa660bf984e: Status 404 returned error can't find the container with id 8cc7a09e488bc539f16e64174a11064532b39d8e50b2183b375c7fa660bf984e
Nov 02 17:31:16.102549 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:16.102518    4456 generic.go:356] "Generic (PLEG): container finished" podID="83deda5b-6ef6-4a66-a76c-888303e6e834" containerID="84c38423cf1378bfec898564bffde08a163aeafa1c31a62db99cbec23b267542" exitCode=0
Nov 02 17:31:16.103044 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:16.102606    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83deda5b-6ef6-4a66-a76c-888303e6e834","Type":"ContainerDied","Data":"84c38423cf1378bfec898564bffde08a163aeafa1c31a62db99cbec23b267542"}
Nov 02 17:31:16.103044 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:16.102679    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83deda5b-6ef6-4a66-a76c-888303e6e834","Type":"ContainerStarted","Data":"8cc7a09e488bc539f16e64174a11064532b39d8e50b2183b375c7fa660bf984e"}
Nov 02 17:31:16.165116 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:31:16.165080    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:31:17.112873 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:17.112835    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83deda5b-6ef6-4a66-a76c-888303e6e834","Type":"ContainerStarted","Data":"41cd825431bee5b7593841b2de296359f8e4e15112a4f2736fd7fd54ca4dd919"}
Nov 02 17:31:17.113352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:17.112909    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83deda5b-6ef6-4a66-a76c-888303e6e834","Type":"ContainerStarted","Data":"5d9893c269c5c9f3e6b32dcb1025e895f19ed5b5612c3006cf80e084840b8809"}
Nov 02 17:31:17.113352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:17.112953    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83deda5b-6ef6-4a66-a76c-888303e6e834","Type":"ContainerStarted","Data":"aabd65ea4fc513f81569cc03f08a9352ade4d6a01ddde7737944ba50b6ca3145"}
Nov 02 17:31:17.113352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:17.112995    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83deda5b-6ef6-4a66-a76c-888303e6e834","Type":"ContainerStarted","Data":"81b88d7707ae8becf555eb774a09f425937113f71a69eaba64d596113df85581"}
Nov 02 17:31:17.113352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:17.113045    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83deda5b-6ef6-4a66-a76c-888303e6e834","Type":"ContainerStarted","Data":"61bf71f81210d36eb8f5bc7972bc216a127f76a287dabc77c73e9fe0b0cc0139"}
Nov 02 17:31:17.113352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:17.113083    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-k8s-0" event={"ID":"83deda5b-6ef6-4a66-a76c-888303e6e834","Type":"ContainerStarted","Data":"bd6105fca3203892e8cc26e61c1ae82d5d6b228d6358c7367e450783ccbc3ce9"}
Nov 02 17:31:20.532887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:20.532848    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:31:32.186587 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:32.186556    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:32.337123 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:32.337087    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:42.096531 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:42.096494    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:42.557766 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:31:42.557737    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:31:46.163765 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:31:46.163726    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:32:15.532899 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:15.532862    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:32:15.548981 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:15.548954    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:32:15.644965 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:15.644918    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-monitoring/prometheus-k8s-0" podStartSLOduration=60.630382469 podStartE2EDuration="1m0.644904612s" totalImagesPullingTime="0s" totalInitContainerRuntime="14.522143ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:31:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:31:17.185634567 +0000 UTC m=+93.676697738" watchObservedRunningTime="2026-11-02 17:32:15.644904612 +0000 UTC m=+152.135967774"
Nov 02 17:32:16.163928 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:32:16.163889    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:32:16.349341 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:16.349316    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-k8s-0"
Nov 02 17:32:17.227120 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:17.227084    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:24.694271 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:24.694234    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:24.898787 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:24.898759    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:25.059662 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:25.059577    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:26.680207 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:26.680162    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:26.836361 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:26.836326    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:28.237177 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:28.237141    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:30.446515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:30.446482    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:30.611152 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:30.611118    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:31.895077 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:31.895045    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:32.088335 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:32.088297    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:32.264102 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:32.264070    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:32.428278 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:32.428244    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:35.652390 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:32:35.652358    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:32:46.162974 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:32:46.162927    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:33:13.626131 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:13.626088    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:16.163659 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:33:16.163622    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:33:23.418793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:23.418759    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:23.754892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:23.754858    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:23.955859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:23.955827    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:24.305540 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.305507    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:24.456163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.456127    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"]
Nov 02 17:33:24.458436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.458415    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.496006 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.495986    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-webhook-cert\""
Nov 02 17:33:24.496328 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.496309    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 17:33:24.496411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.496378    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-nv7lh\""
Nov 02 17:33:24.508923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.508907    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 17:33:24.515315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.515296    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rnnmb\" (UniqueName: \"kubernetes.io/projected/ec905c34-6dfb-483f-bb3d-7ba412262584-kube-api-access-rnnmb\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.515492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.515345    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec905c34-6dfb-483f-bb3d-7ba412262584-metrics-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.515492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.515370    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ec905c34-6dfb-483f-bb3d-7ba412262584-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.515608 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.515489    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec905c34-6dfb-483f-bb3d-7ba412262584-tmp\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.515608 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.515518    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ec905c34-6dfb-483f-bb3d-7ba412262584-manifests\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.515715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.515651    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec905c34-6dfb-483f-bb3d-7ba412262584-webhook-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.519475 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.519453    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"]
Nov 02 17:33:24.544184 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.544164    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 17:33:24.544438 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.544419    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 17:33:24.561898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.561833    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:24.616393 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.616362    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rnnmb\" (UniqueName: \"kubernetes.io/projected/ec905c34-6dfb-483f-bb3d-7ba412262584-kube-api-access-rnnmb\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.616537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.616412    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec905c34-6dfb-483f-bb3d-7ba412262584-metrics-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.616537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.616522    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ec905c34-6dfb-483f-bb3d-7ba412262584-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.616648 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.616583    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec905c34-6dfb-483f-bb3d-7ba412262584-tmp\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.616648 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.616605    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ec905c34-6dfb-483f-bb3d-7ba412262584-manifests\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.616648 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.616645    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec905c34-6dfb-483f-bb3d-7ba412262584-webhook-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.617029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.617001    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manifests\" (UniqueName: \"kubernetes.io/empty-dir/ec905c34-6dfb-483f-bb3d-7ba412262584-manifests\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.619140 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.619114    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/ec905c34-6dfb-483f-bb3d-7ba412262584-tmp\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.619223 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.619211    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"webhook-cert\" (UniqueName: \"kubernetes.io/secret/ec905c34-6dfb-483f-bb3d-7ba412262584-webhook-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.619308 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.619289    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-cert\" (UniqueName: \"kubernetes.io/secret/ec905c34-6dfb-483f-bb3d-7ba412262584-metrics-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.619308 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.619301    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"apiservice-cert\" (UniqueName: \"kubernetes.io/secret/ec905c34-6dfb-483f-bb3d-7ba412262584-apiservice-cert\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.659208 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.659186    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rnnmb\" (UniqueName: \"kubernetes.io/projected/ec905c34-6dfb-483f-bb3d-7ba412262584-kube-api-access-rnnmb\") pod \"opendatahub-operator-controller-manager-76f76b79c-7n462\" (UID: \"ec905c34-6dfb-483f-bb3d-7ba412262584\") " pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.770539 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.770510    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:24.776387 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.776360    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:24.923865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:24.923832    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:25.078784 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:25.078710    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:25.133568 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:25.133547    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"]
Nov 02 17:33:25.135540 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:33:25.135512    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec905c34_6dfb_483f_bb3d_7ba412262584.slice/crio-29f7a02c29f1bb136e2bdfbf148a21d3505b02594e8f698d79d1c3f778a8bde8 WatchSource:0}: Error finding container 29f7a02c29f1bb136e2bdfbf148a21d3505b02594e8f698d79d1c3f778a8bde8: Status 404 returned error can't find the container with id 29f7a02c29f1bb136e2bdfbf148a21d3505b02594e8f698d79d1c3f778a8bde8
Nov 02 17:33:25.580707 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:25.580671    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" event={"ID":"ec905c34-6dfb-483f-bb3d-7ba412262584","Type":"ContainerStarted","Data":"29f7a02c29f1bb136e2bdfbf148a21d3505b02594e8f698d79d1c3f778a8bde8"}
Nov 02 17:33:27.246468 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:27.246423    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:27.435696 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:27.435656    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:27.588311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:27.588221    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:28.601076 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:28.601041    4456 generic.go:356] "Generic (PLEG): container finished" podID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerID="8522bae933b717a597952f1fdab710f757ca210de700bebeef70d46cbb4dfe3b" exitCode=0
Nov 02 17:33:28.601536 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:28.601110    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" event={"ID":"ec905c34-6dfb-483f-bb3d-7ba412262584","Type":"ContainerDied","Data":"8522bae933b717a597952f1fdab710f757ca210de700bebeef70d46cbb4dfe3b"}
Nov 02 17:33:29.611074 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:29.611036    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" event={"ID":"ec905c34-6dfb-483f-bb3d-7ba412262584","Type":"ContainerStarted","Data":"98bb1cdd787c025aa16ef0cb488e4b052f1a1c499a91687b7d53b720295e0af2"}
Nov 02 17:33:29.611597 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:29.611095    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:29.697576 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:29.697505    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podStartSLOduration=2.037669745 podStartE2EDuration="5.697485836s" totalImagesPullingTime="3.191412545s" totalInitContainerRuntime="468.403546ms" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:24 +0000 UTC" imagePullSessionsCount=2 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:29.684318549 +0000 UTC m=+226.175381721" watchObservedRunningTime="2026-11-02 17:33:29.697485836 +0000 UTC m=+226.188548999"
Nov 02 17:33:32.445708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:32.445669    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:32.618614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:32.618581    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:32.791916 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:32.791791    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:39.015891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:39.015831    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:40.613894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:40.613868    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 17:33:42.599182 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:42.599140    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:43.259382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:43.259349    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:44.076633 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:44.076595    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:44.597946 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:44.597912    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:46.163712 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:33:46.163670    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:33:48.655231 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:48.655196    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:50.253156 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:50.253116    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:50.568172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:50.568094    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:50.911421 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:50.911343    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:51.133650 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:51.133614    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:51.311173 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:51.311131    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:51.475577 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:51.475544    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:53.060446 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:53.060370    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:53.238238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:53.238203    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:55.234528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.234484    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:55.405940 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.405905    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9"]
Nov 02 17:33:55.409769 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.409748    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9"
Nov 02 17:33:55.427488 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.427461    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9"]
Nov 02 17:33:55.428863 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.428839    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:55.438675 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.438657    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 17:33:55.438764 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.438685    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-x9lxq\""
Nov 02 17:33:55.438860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.438846    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 17:33:55.486810 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.486739    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5fbjf\" (UniqueName: \"kubernetes.io/projected/9fe1660e-3733-4c7f-bc5c-3820d84aa143-kube-api-access-5fbjf\") pod \"cert-manager-cainjector-5c57bbd4fc-9mjv9\" (UID: \"9fe1660e-3733-4c7f-bc5c-3820d84aa143\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9"
Nov 02 17:33:55.486810 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.486789    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fe1660e-3733-4c7f-bc5c-3820d84aa143-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9mjv9\" (UID: \"9fe1660e-3733-4c7f-bc5c-3820d84aa143\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9"
Nov 02 17:33:55.587504 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.587470    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5fbjf\" (UniqueName: \"kubernetes.io/projected/9fe1660e-3733-4c7f-bc5c-3820d84aa143-kube-api-access-5fbjf\") pod \"cert-manager-cainjector-5c57bbd4fc-9mjv9\" (UID: \"9fe1660e-3733-4c7f-bc5c-3820d84aa143\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9"
Nov 02 17:33:55.587675 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.587522    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fe1660e-3733-4c7f-bc5c-3820d84aa143-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9mjv9\" (UID: \"9fe1660e-3733-4c7f-bc5c-3820d84aa143\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9"
Nov 02 17:33:55.603290 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.603256    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:55.611500 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.611477    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/9fe1660e-3733-4c7f-bc5c-3820d84aa143-bound-sa-token\") pod \"cert-manager-cainjector-5c57bbd4fc-9mjv9\" (UID: \"9fe1660e-3733-4c7f-bc5c-3820d84aa143\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9"
Nov 02 17:33:55.614798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.614774    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5fbjf\" (UniqueName: \"kubernetes.io/projected/9fe1660e-3733-4c7f-bc5c-3820d84aa143-kube-api-access-5fbjf\") pod \"cert-manager-cainjector-5c57bbd4fc-9mjv9\" (UID: \"9fe1660e-3733-4c7f-bc5c-3820d84aa143\") " pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9"
Nov 02 17:33:55.734059 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.734028    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9"
Nov 02 17:33:55.783995 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:55.782625    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:56.038955 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:56.038932    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9"]
Nov 02 17:33:56.041184 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:33:56.041158    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod9fe1660e_3733_4c7f_bc5c_3820d84aa143.slice/crio-208269ec30d2c6bb80b935d0fc188903a89a496be57e303855dfac7a7c20332e WatchSource:0}: Error finding container 208269ec30d2c6bb80b935d0fc188903a89a496be57e303855dfac7a7c20332e: Status 404 returned error can't find the container with id 208269ec30d2c6bb80b935d0fc188903a89a496be57e303855dfac7a7c20332e
Nov 02 17:33:56.715218 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:56.715174    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9" event={"ID":"9fe1660e-3733-4c7f-bc5c-3820d84aa143","Type":"ContainerStarted","Data":"208269ec30d2c6bb80b935d0fc188903a89a496be57e303855dfac7a7c20332e"}
Nov 02 17:33:57.280066 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.280030    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7"]
Nov 02 17:33:57.284759 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.284731    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7"
Nov 02 17:33:57.303829 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.303783    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/08c9aa32-dadc-4112-b608-4cc64f56295c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-c8vr7\" (UID: \"08c9aa32-dadc-4112-b608-4cc64f56295c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7"
Nov 02 17:33:57.303962 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.303853    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/08c9aa32-dadc-4112-b608-4cc64f56295c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-c8vr7\" (UID: \"08c9aa32-dadc-4112-b608-4cc64f56295c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7"
Nov 02 17:33:57.303962 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.303924    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2md2\" (UniqueName: \"kubernetes.io/projected/08c9aa32-dadc-4112-b608-4cc64f56295c-kube-api-access-n2md2\") pod \"kuadrant-console-plugin-66c4fb575d-c8vr7\" (UID: \"08c9aa32-dadc-4112-b608-4cc64f56295c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7"
Nov 02 17:33:57.318684 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.318659    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7"]
Nov 02 17:33:57.328053 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.328025    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 02 17:33:57.328053 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.328054    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 02 17:33:57.328235 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.328092    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 17:33:57.328235 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.328116    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 17:33:57.329480 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.329456    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p5nzm\""
Nov 02 17:33:57.340335 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.340308    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:57.404660 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.404620    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/08c9aa32-dadc-4112-b608-4cc64f56295c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-c8vr7\" (UID: \"08c9aa32-dadc-4112-b608-4cc64f56295c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7"
Nov 02 17:33:57.404864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.404675    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/08c9aa32-dadc-4112-b608-4cc64f56295c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-c8vr7\" (UID: \"08c9aa32-dadc-4112-b608-4cc64f56295c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7"
Nov 02 17:33:57.404864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.404706    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2md2\" (UniqueName: \"kubernetes.io/projected/08c9aa32-dadc-4112-b608-4cc64f56295c-kube-api-access-n2md2\") pod \"kuadrant-console-plugin-66c4fb575d-c8vr7\" (UID: \"08c9aa32-dadc-4112-b608-4cc64f56295c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7"
Nov 02 17:33:57.405561 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.405529    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"nginx-conf\" (UniqueName: \"kubernetes.io/configmap/08c9aa32-dadc-4112-b608-4cc64f56295c-nginx-conf\") pod \"kuadrant-console-plugin-66c4fb575d-c8vr7\" (UID: \"08c9aa32-dadc-4112-b608-4cc64f56295c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7"
Nov 02 17:33:57.408023 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.408000    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugin-serving-cert\" (UniqueName: \"kubernetes.io/secret/08c9aa32-dadc-4112-b608-4cc64f56295c-plugin-serving-cert\") pod \"kuadrant-console-plugin-66c4fb575d-c8vr7\" (UID: \"08c9aa32-dadc-4112-b608-4cc64f56295c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7"
Nov 02 17:33:57.429574 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.429550    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2md2\" (UniqueName: \"kubernetes.io/projected/08c9aa32-dadc-4112-b608-4cc64f56295c-kube-api-access-n2md2\") pod \"kuadrant-console-plugin-66c4fb575d-c8vr7\" (UID: \"08c9aa32-dadc-4112-b608-4cc64f56295c\") " pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7"
Nov 02 17:33:57.511330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.511232    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:57.600071 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.600002    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7"
Nov 02 17:33:57.919392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:57.919327    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7"]
Nov 02 17:33:57.923164 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:33:57.923130    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod08c9aa32_dadc_4112_b608_4cc64f56295c.slice/crio-a2bdfced8b3538ff2ed8dce83b60c4df3d33683656e4e76109ddd37947a13e3b WatchSource:0}: Error finding container a2bdfced8b3538ff2ed8dce83b60c4df3d33683656e4e76109ddd37947a13e3b: Status 404 returned error can't find the container with id a2bdfced8b3538ff2ed8dce83b60c4df3d33683656e4e76109ddd37947a13e3b
Nov 02 17:33:58.726397 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:58.726343    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7" event={"ID":"08c9aa32-dadc-4112-b608-4cc64f56295c","Type":"ContainerStarted","Data":"a2bdfced8b3538ff2ed8dce83b60c4df3d33683656e4e76109ddd37947a13e3b"}
Nov 02 17:33:58.825557 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:58.825448    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:59.006703 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.006424    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:59.267501 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.267426    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:59.570859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.570769    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/kube-auth-proxy-78648788bf-npsk7"]
Nov 02 17:33:59.574276 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.574259    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78648788bf-npsk7"
Nov 02 17:33:59.600070 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.600037    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78648788bf-npsk7"]
Nov 02 17:33:59.609328 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.609305    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-creds\""
Nov 02 17:33:59.609449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.609346    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 17:33:59.610726 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.610701    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4679w\""
Nov 02 17:33:59.610922 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.610908    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 17:33:59.610987 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.610964    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 17:33:59.629460 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.629430    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/347c8613-0791-494e-83e3-faded5593d88-tmp\") pod \"kube-auth-proxy-78648788bf-npsk7\" (UID: \"347c8613-0791-494e-83e3-faded5593d88\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-npsk7"
Nov 02 17:33:59.629614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.629531    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347c8613-0791-494e-83e3-faded5593d88-tls-certs\") pod \"kube-auth-proxy-78648788bf-npsk7\" (UID: \"347c8613-0791-494e-83e3-faded5593d88\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-npsk7"
Nov 02 17:33:59.629614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.629564    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2sfc\" (UniqueName: \"kubernetes.io/projected/347c8613-0791-494e-83e3-faded5593d88-kube-api-access-z2sfc\") pod \"kube-auth-proxy-78648788bf-npsk7\" (UID: \"347c8613-0791-494e-83e3-faded5593d88\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-npsk7"
Nov 02 17:33:59.648920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.648860    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:59.725641 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.725607    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"]
Nov 02 17:33:59.730236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.729647    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"
Nov 02 17:33:59.730236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.730012    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347c8613-0791-494e-83e3-faded5593d88-tls-certs\") pod \"kube-auth-proxy-78648788bf-npsk7\" (UID: \"347c8613-0791-494e-83e3-faded5593d88\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-npsk7"
Nov 02 17:33:59.730236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.730057    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2sfc\" (UniqueName: \"kubernetes.io/projected/347c8613-0791-494e-83e3-faded5593d88-kube-api-access-z2sfc\") pod \"kube-auth-proxy-78648788bf-npsk7\" (UID: \"347c8613-0791-494e-83e3-faded5593d88\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-npsk7"
Nov 02 17:33:59.730236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.730122    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/347c8613-0791-494e-83e3-faded5593d88-tmp\") pod \"kube-auth-proxy-78648788bf-npsk7\" (UID: \"347c8613-0791-494e-83e3-faded5593d88\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-npsk7"
Nov 02 17:33:59.732932 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.732909    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/347c8613-0791-494e-83e3-faded5593d88-tmp\") pod \"kube-auth-proxy-78648788bf-npsk7\" (UID: \"347c8613-0791-494e-83e3-faded5593d88\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-npsk7"
Nov 02 17:33:59.733464 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.733395    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/347c8613-0791-494e-83e3-faded5593d88-tls-certs\") pod \"kube-auth-proxy-78648788bf-npsk7\" (UID: \"347c8613-0791-494e-83e3-faded5593d88\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-npsk7"
Nov 02 17:33:59.733837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.733788    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9" event={"ID":"9fe1660e-3733-4c7f-bc5c-3820d84aa143","Type":"ContainerStarted","Data":"cba9f82f9983882e3976ebd947a2c9e62cc786d2b9bba3e2b09311bc77e055cb"}
Nov 02 17:33:59.749215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.749151    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"]
Nov 02 17:33:59.754618 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.754595    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"limitador-limits-config-limitador\""
Nov 02 17:33:59.772098 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.772071    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2sfc\" (UniqueName: \"kubernetes.io/projected/347c8613-0791-494e-83e3-faded5593d88-kube-api-access-z2sfc\") pod \"kube-auth-proxy-78648788bf-npsk7\" (UID: \"347c8613-0791-494e-83e3-faded5593d88\") " pod="openshift-ingress/kube-auth-proxy-78648788bf-npsk7"
Nov 02 17:33:59.831258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.831172    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56369ef1-5944-4615-9baa-ca41c1f65b7e-config-file\") pod \"limitador-limitador-74dfcfb894-tbfm8\" (UID: \"56369ef1-5944-4615-9baa-ca41c1f65b7e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"
Nov 02 17:33:59.831558 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.831373    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gpc9l\" (UniqueName: \"kubernetes.io/projected/56369ef1-5944-4615-9baa-ca41c1f65b7e-kube-api-access-gpc9l\") pod \"limitador-limitador-74dfcfb894-tbfm8\" (UID: \"56369ef1-5944-4615-9baa-ca41c1f65b7e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"
Nov 02 17:33:59.833138 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.833093    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9" podStartSLOduration=1.452386837 podStartE2EDuration="4.833078301s" totalImagesPullingTime="3.380691464s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:33:59.832229887 +0000 UTC m=+256.323293062" watchObservedRunningTime="2026-11-02 17:33:59.833078301 +0000 UTC m=+256.324141462"
Nov 02 17:33:59.860349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.860286    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:33:59.864864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.864834    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"]
Nov 02 17:33:59.888983 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.888954    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/kube-auth-proxy-78648788bf-npsk7"
Nov 02 17:33:59.932207 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.932171    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gpc9l\" (UniqueName: \"kubernetes.io/projected/56369ef1-5944-4615-9baa-ca41c1f65b7e-kube-api-access-gpc9l\") pod \"limitador-limitador-74dfcfb894-tbfm8\" (UID: \"56369ef1-5944-4615-9baa-ca41c1f65b7e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"
Nov 02 17:33:59.932349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.932297    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56369ef1-5944-4615-9baa-ca41c1f65b7e-config-file\") pod \"limitador-limitador-74dfcfb894-tbfm8\" (UID: \"56369ef1-5944-4615-9baa-ca41c1f65b7e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"
Nov 02 17:33:59.933071 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.933040    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56369ef1-5944-4615-9baa-ca41c1f65b7e-config-file\") pod \"limitador-limitador-74dfcfb894-tbfm8\" (UID: \"56369ef1-5944-4615-9baa-ca41c1f65b7e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"
Nov 02 17:33:59.970385 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:33:59.970281    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gpc9l\" (UniqueName: \"kubernetes.io/projected/56369ef1-5944-4615-9baa-ca41c1f65b7e-kube-api-access-gpc9l\") pod \"limitador-limitador-74dfcfb894-tbfm8\" (UID: \"56369ef1-5944-4615-9baa-ca41c1f65b7e\") " pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"
Nov 02 17:34:00.058446 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:00.058408    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"
Nov 02 17:34:00.064920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:00.064893    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:00.210732 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:00.210686    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/kube-auth-proxy-78648788bf-npsk7"]
Nov 02 17:34:00.256084 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:00.256048    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:00.372884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:00.372779    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"]
Nov 02 17:34:00.375608 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:34:00.375581    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56369ef1_5944_4615_9baa_ca41c1f65b7e.slice/crio-0e472ce73c453c324112c8124b45be4771351ce9994deb19b61aa2dba9db6cf3 WatchSource:0}: Error finding container 0e472ce73c453c324112c8124b45be4771351ce9994deb19b61aa2dba9db6cf3: Status 404 returned error can't find the container with id 0e472ce73c453c324112c8124b45be4771351ce9994deb19b61aa2dba9db6cf3
Nov 02 17:34:00.744922 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:00.744885    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:00.746102 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:00.746072    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78648788bf-npsk7" event={"ID":"347c8613-0791-494e-83e3-faded5593d88","Type":"ContainerStarted","Data":"da4ca04fea694213296eb9a7bb7073736e98ef3ef73c1f9e226d785a4078311d"}
Nov 02 17:34:00.747689 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:00.747647    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8" event={"ID":"56369ef1-5944-4615-9baa-ca41c1f65b7e","Type":"ContainerStarted","Data":"0e472ce73c453c324112c8124b45be4771351ce9994deb19b61aa2dba9db6cf3"}
Nov 02 17:34:00.822045 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:00.822017    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wjh6s"]
Nov 02 17:34:00.825561 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:00.825540    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wjh6s"
Nov 02 17:34:00.845455 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:00.845423    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wjh6s"]
Nov 02 17:34:00.919654 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:00.919614    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kqm5s\""
Nov 02 17:34:00.940121 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:00.940086    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw4rd\" (UniqueName: \"kubernetes.io/projected/6fc0745b-20b4-40ee-aed8-be6323f7724f-kube-api-access-gw4rd\") pod \"authorino-57f7f8659f-wjh6s\" (UID: \"6fc0745b-20b4-40ee-aed8-be6323f7724f\") " pod="kuadrant-system/authorino-57f7f8659f-wjh6s"
Nov 02 17:34:00.945404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:00.945347    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:01.040887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:01.040792    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw4rd\" (UniqueName: \"kubernetes.io/projected/6fc0745b-20b4-40ee-aed8-be6323f7724f-kube-api-access-gw4rd\") pod \"authorino-57f7f8659f-wjh6s\" (UID: \"6fc0745b-20b4-40ee-aed8-be6323f7724f\") " pod="kuadrant-system/authorino-57f7f8659f-wjh6s"
Nov 02 17:34:01.087800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:01.087744    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw4rd\" (UniqueName: \"kubernetes.io/projected/6fc0745b-20b4-40ee-aed8-be6323f7724f-kube-api-access-gw4rd\") pod \"authorino-57f7f8659f-wjh6s\" (UID: \"6fc0745b-20b4-40ee-aed8-be6323f7724f\") " pod="kuadrant-system/authorino-57f7f8659f-wjh6s"
Nov 02 17:34:01.138119 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:01.138086    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wjh6s"
Nov 02 17:34:01.150796 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:01.150688    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:01.353362 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:01.353266    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:01.479692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:01.479651    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wjh6s"]
Nov 02 17:34:01.486596 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:34:01.486566    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod6fc0745b_20b4_40ee_aed8_be6323f7724f.slice/crio-787d886c6b8702d043d0b798978ed0666a44b7f65ab19e000408a96197df610e WatchSource:0}: Error finding container 787d886c6b8702d043d0b798978ed0666a44b7f65ab19e000408a96197df610e: Status 404 returned error can't find the container with id 787d886c6b8702d043d0b798978ed0666a44b7f65ab19e000408a96197df610e
Nov 02 17:34:01.755735 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:01.755697    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wjh6s" event={"ID":"6fc0745b-20b4-40ee-aed8-be6323f7724f","Type":"ContainerStarted","Data":"787d886c6b8702d043d0b798978ed0666a44b7f65ab19e000408a96197df610e"}
Nov 02 17:34:02.822873 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:02.822776    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:03.028628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:03.028176    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:03.429842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:03.429791    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["cert-manager/cert-manager-597b95cb84-hr8vz"]
Nov 02 17:34:03.435164 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:03.435142    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hr8vz"
Nov 02 17:34:03.462299 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:03.461259    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hr8vz"]
Nov 02 17:34:03.474462 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:03.464584    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-84fvd\""
Nov 02 17:34:03.474462 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:03.465035    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e73be583-2a26-4379-8169-0a6b8784a11b-bound-sa-token\") pod \"cert-manager-597b95cb84-hr8vz\" (UID: \"e73be583-2a26-4379-8169-0a6b8784a11b\") " pod="cert-manager/cert-manager-597b95cb84-hr8vz"
Nov 02 17:34:03.474462 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:03.465074    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8rts5\" (UniqueName: \"kubernetes.io/projected/e73be583-2a26-4379-8169-0a6b8784a11b-kube-api-access-8rts5\") pod \"cert-manager-597b95cb84-hr8vz\" (UID: \"e73be583-2a26-4379-8169-0a6b8784a11b\") " pod="cert-manager/cert-manager-597b95cb84-hr8vz"
Nov 02 17:34:03.565647 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:03.565607    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8rts5\" (UniqueName: \"kubernetes.io/projected/e73be583-2a26-4379-8169-0a6b8784a11b-kube-api-access-8rts5\") pod \"cert-manager-597b95cb84-hr8vz\" (UID: \"e73be583-2a26-4379-8169-0a6b8784a11b\") " pod="cert-manager/cert-manager-597b95cb84-hr8vz"
Nov 02 17:34:03.565875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:03.565757    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e73be583-2a26-4379-8169-0a6b8784a11b-bound-sa-token\") pod \"cert-manager-597b95cb84-hr8vz\" (UID: \"e73be583-2a26-4379-8169-0a6b8784a11b\") " pod="cert-manager/cert-manager-597b95cb84-hr8vz"
Nov 02 17:34:03.591028 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:03.590965    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8rts5\" (UniqueName: \"kubernetes.io/projected/e73be583-2a26-4379-8169-0a6b8784a11b-kube-api-access-8rts5\") pod \"cert-manager-597b95cb84-hr8vz\" (UID: \"e73be583-2a26-4379-8169-0a6b8784a11b\") " pod="cert-manager/cert-manager-597b95cb84-hr8vz"
Nov 02 17:34:03.595268 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:03.595213    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"bound-sa-token\" (UniqueName: \"kubernetes.io/projected/e73be583-2a26-4379-8169-0a6b8784a11b-bound-sa-token\") pod \"cert-manager-597b95cb84-hr8vz\" (UID: \"e73be583-2a26-4379-8169-0a6b8784a11b\") " pod="cert-manager/cert-manager-597b95cb84-hr8vz"
Nov 02 17:34:03.750700 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:03.750641    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="cert-manager/cert-manager-597b95cb84-hr8vz"
Nov 02 17:34:04.098905 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:04.098843    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["cert-manager/cert-manager-597b95cb84-hr8vz"]
Nov 02 17:34:04.103428 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:34:04.103404    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73be583_2a26_4379_8169_0a6b8784a11b.slice/crio-c8c29e1724dfe973e6ebbafe2d26fac29415c94500637d1159cc56fd690f19d9 WatchSource:0}: Error finding container c8c29e1724dfe973e6ebbafe2d26fac29415c94500637d1159cc56fd690f19d9: Status 404 returned error can't find the container with id c8c29e1724dfe973e6ebbafe2d26fac29415c94500637d1159cc56fd690f19d9
Nov 02 17:34:04.271847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:04.269071    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:04.778344 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:04.778304    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8" event={"ID":"56369ef1-5944-4615-9baa-ca41c1f65b7e","Type":"ContainerStarted","Data":"54255a4cb5d88aea6a8ad6044202385653468b907f2c6a818abb818254f23d32"}
Nov 02 17:34:04.778548 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:04.778385    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"
Nov 02 17:34:04.780020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:04.779983    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/kube-auth-proxy-78648788bf-npsk7" event={"ID":"347c8613-0791-494e-83e3-faded5593d88","Type":"ContainerStarted","Data":"52c0f7af1079df1129720e742839ea3c4f8d085457476e96246be042d96f837c"}
Nov 02 17:34:04.781955 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:04.781928    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hr8vz" event={"ID":"e73be583-2a26-4379-8169-0a6b8784a11b","Type":"ContainerStarted","Data":"f102af8d8427f21ccd5327dba082268df8744b692aa3dbb2752b19099048e51c"}
Nov 02 17:34:04.782061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:04.781966    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hr8vz" event={"ID":"e73be583-2a26-4379-8169-0a6b8784a11b","Type":"ContainerStarted","Data":"c8c29e1724dfe973e6ebbafe2d26fac29415c94500637d1159cc56fd690f19d9"}
Nov 02 17:34:04.820031 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:04.819970    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8" podStartSLOduration=2.110985563 podStartE2EDuration="5.819957403s" totalImagesPullingTime="3.70897184s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:04.819364926 +0000 UTC m=+261.310428087" watchObservedRunningTime="2026-11-02 17:34:04.819957403 +0000 UTC m=+261.311020593"
Nov 02 17:34:04.865056 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:04.865004    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="cert-manager/cert-manager-597b95cb84-hr8vz" podStartSLOduration=1.864991015 podStartE2EDuration="1.864991015s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:04.863926252 +0000 UTC m=+261.354989423" watchObservedRunningTime="2026-11-02 17:34:04.864991015 +0000 UTC m=+261.356054175"
Nov 02 17:34:05.323379 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:05.322765    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:05.614839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:05.614734    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:05.785596 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:05.785561    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:05.787561 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:05.787535    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7" event={"ID":"08c9aa32-dadc-4112-b608-4cc64f56295c","Type":"ContainerStarted","Data":"7efdd143891e4950ddfda0399cb2ffb47f403b3cd9801b880307c7f4f3e1e4d8"}
Nov 02 17:34:05.788994 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:05.788968    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wjh6s" event={"ID":"6fc0745b-20b4-40ee-aed8-be6323f7724f","Type":"ContainerStarted","Data":"35117c7c4acc0c95a920c168e58d820808f93cb226524f283e28fadb82f4192e"}
Nov 02 17:34:05.828620 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:05.828568    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7" podStartSLOduration=1.484529474 podStartE2EDuration="8.828554996s" totalImagesPullingTime="7.344025522s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:05.828087744 +0000 UTC m=+262.319150934" watchObservedRunningTime="2026-11-02 17:34:05.828554996 +0000 UTC m=+262.319618201"
Nov 02 17:34:05.829624 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:05.829593    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/kube-auth-proxy-78648788bf-npsk7" podStartSLOduration=2.894216558 podStartE2EDuration="6.829584379s" totalImagesPullingTime="3.935367821s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:33:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:04.910473988 +0000 UTC m=+261.401537148" watchObservedRunningTime="2026-11-02 17:34:05.829584379 +0000 UTC m=+262.320647577"
Nov 02 17:34:05.879307 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:05.879202    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-57f7f8659f-wjh6s" podStartSLOduration=2.100223749 podStartE2EDuration="5.879188491s" totalImagesPullingTime="3.778964742s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:00 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:05.876991215 +0000 UTC m=+262.368054396" watchObservedRunningTime="2026-11-02 17:34:05.879188491 +0000 UTC m=+262.370251651"
Nov 02 17:34:05.973324 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:05.973289    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"]
Nov 02 17:34:05.977097 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:05.977079    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 17:34:06.007448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:06.007425    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p2jm6\""
Nov 02 17:34:06.017611 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:06.017460    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"]
Nov 02 17:34:06.088765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:06.088726    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1ea3dac-7073-4564-b247-89ea94a64342-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6jpq8\" (UID: \"a1ea3dac-7073-4564-b247-89ea94a64342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 17:34:06.088765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:06.088769    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s5bh4\" (UniqueName: \"kubernetes.io/projected/a1ea3dac-7073-4564-b247-89ea94a64342-kube-api-access-s5bh4\") pod \"kuadrant-operator-controller-manager-678d87766f-6jpq8\" (UID: \"a1ea3dac-7073-4564-b247-89ea94a64342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 17:34:06.189609 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:06.189574    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1ea3dac-7073-4564-b247-89ea94a64342-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6jpq8\" (UID: \"a1ea3dac-7073-4564-b247-89ea94a64342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 17:34:06.189789 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:06.189632    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s5bh4\" (UniqueName: \"kubernetes.io/projected/a1ea3dac-7073-4564-b247-89ea94a64342-kube-api-access-s5bh4\") pod \"kuadrant-operator-controller-manager-678d87766f-6jpq8\" (UID: \"a1ea3dac-7073-4564-b247-89ea94a64342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 17:34:06.189976 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:06.189953    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1ea3dac-7073-4564-b247-89ea94a64342-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-6jpq8\" (UID: \"a1ea3dac-7073-4564-b247-89ea94a64342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 17:34:06.217934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:06.217905    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s5bh4\" (UniqueName: \"kubernetes.io/projected/a1ea3dac-7073-4564-b247-89ea94a64342-kube-api-access-s5bh4\") pod \"kuadrant-operator-controller-manager-678d87766f-6jpq8\" (UID: \"a1ea3dac-7073-4564-b247-89ea94a64342\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 17:34:06.289691 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:06.289660    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 17:34:06.624958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:06.624878    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"]
Nov 02 17:34:06.641157 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:34:06.641127    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ea3dac_7073_4564_b247_89ea94a64342.slice/crio-46103e666e23a14578de5271d8da8f28533195c6f4d5e5ef5cdbdc44d4803cf3 WatchSource:0}: Error finding container 46103e666e23a14578de5271d8da8f28533195c6f4d5e5ef5cdbdc44d4803cf3: Status 404 returned error can't find the container with id 46103e666e23a14578de5271d8da8f28533195c6f4d5e5ef5cdbdc44d4803cf3
Nov 02 17:34:06.794032 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:06.793992    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" event={"ID":"a1ea3dac-7073-4564-b247-89ea94a64342","Type":"ContainerStarted","Data":"46103e666e23a14578de5271d8da8f28533195c6f4d5e5ef5cdbdc44d4803cf3"}
Nov 02 17:34:07.654287 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:07.654252    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wjh6s"]
Nov 02 17:34:07.798877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:07.798838    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/authorino-57f7f8659f-wjh6s" podUID="6fc0745b-20b4-40ee-aed8-be6323f7724f" containerName="authorino" containerID="cri-o://35117c7c4acc0c95a920c168e58d820808f93cb226524f283e28fadb82f4192e" gracePeriod=30
Nov 02 17:34:08.062369 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:08.062338    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wjh6s"
Nov 02 17:34:08.106630 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:08.106589    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6fc0745b-20b4-40ee-aed8-be6323f7724f-kube-api-access-gw4rd\" (UniqueName: \"kubernetes.io/projected/6fc0745b-20b4-40ee-aed8-be6323f7724f-kube-api-access-gw4rd\") pod \"6fc0745b-20b4-40ee-aed8-be6323f7724f\" (UID: \"6fc0745b-20b4-40ee-aed8-be6323f7724f\") "
Nov 02 17:34:08.109173 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:08.109145    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6fc0745b-20b4-40ee-aed8-be6323f7724f-kube-api-access-gw4rd" pod "6fc0745b-20b4-40ee-aed8-be6323f7724f" (UID: "6fc0745b-20b4-40ee-aed8-be6323f7724f"). InnerVolumeSpecName "kube-api-access-gw4rd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:34:08.207338 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:08.207298    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw4rd\" (UniqueName: \"kubernetes.io/projected/6fc0745b-20b4-40ee-aed8-be6323f7724f-kube-api-access-gw4rd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:34:08.805268 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:08.805229    4456 generic.go:356] "Generic (PLEG): container finished" podID="6fc0745b-20b4-40ee-aed8-be6323f7724f" containerID="35117c7c4acc0c95a920c168e58d820808f93cb226524f283e28fadb82f4192e" exitCode=0
Nov 02 17:34:08.805770 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:08.805290    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-57f7f8659f-wjh6s"
Nov 02 17:34:08.805770 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:08.805317    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wjh6s" event={"ID":"6fc0745b-20b4-40ee-aed8-be6323f7724f","Type":"ContainerDied","Data":"35117c7c4acc0c95a920c168e58d820808f93cb226524f283e28fadb82f4192e"}
Nov 02 17:34:08.805770 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:08.805366    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-57f7f8659f-wjh6s" event={"ID":"6fc0745b-20b4-40ee-aed8-be6323f7724f","Type":"ContainerDied","Data":"787d886c6b8702d043d0b798978ed0666a44b7f65ab19e000408a96197df610e"}
Nov 02 17:34:08.805770 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:08.805385    4456 scope.go:122] "RemoveContainer" containerID="35117c7c4acc0c95a920c168e58d820808f93cb226524f283e28fadb82f4192e"
Nov 02 17:34:08.816177 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:08.816150    4456 scope.go:122] "RemoveContainer" containerID="35117c7c4acc0c95a920c168e58d820808f93cb226524f283e28fadb82f4192e"
Nov 02 17:34:08.816883 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:34:08.816857    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"35117c7c4acc0c95a920c168e58d820808f93cb226524f283e28fadb82f4192e\": container with ID starting with 35117c7c4acc0c95a920c168e58d820808f93cb226524f283e28fadb82f4192e not found: ID does not exist" containerID="35117c7c4acc0c95a920c168e58d820808f93cb226524f283e28fadb82f4192e"
Nov 02 17:34:08.816998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:08.816895    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"35117c7c4acc0c95a920c168e58d820808f93cb226524f283e28fadb82f4192e"} err="failed to get container status \"35117c7c4acc0c95a920c168e58d820808f93cb226524f283e28fadb82f4192e\": rpc error: code = NotFound desc = could not find container \"35117c7c4acc0c95a920c168e58d820808f93cb226524f283e28fadb82f4192e\": container with ID starting with 35117c7c4acc0c95a920c168e58d820808f93cb226524f283e28fadb82f4192e not found: ID does not exist"
Nov 02 17:34:08.857753 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:08.857712    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wjh6s"]
Nov 02 17:34:08.870011 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:08.869977    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-57f7f8659f-wjh6s"]
Nov 02 17:34:09.613741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:09.613707    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6fc0745b-20b4-40ee-aed8-be6323f7724f" path="/var/lib/kubelet/pods/6fc0745b-20b4-40ee-aed8-be6323f7724f/volumes"
Nov 02 17:34:10.818317 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:10.818284    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" event={"ID":"a1ea3dac-7073-4564-b247-89ea94a64342","Type":"ContainerStarted","Data":"cdd773afd598fd21cda432d2c443c479f7282ad8cf4c3c694738ed4f0698ba92"}
Nov 02 17:34:10.818712 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:10.818426    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 17:34:10.889158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:10.889106    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podStartSLOduration=2.144515742 podStartE2EDuration="5.889090091s" totalImagesPullingTime="3.744574349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:05 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:10.875884519 +0000 UTC m=+267.366947691" watchObservedRunningTime="2026-11-02 17:34:10.889090091 +0000 UTC m=+267.380153253"
Nov 02 17:34:13.401302 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:13.401272    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:13.550306 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:13.550271    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:15.790568 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:15.790537    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"
Nov 02 17:34:15.854610 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:15.854574    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:16.006677 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:16.006641    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:16.163642 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:34:16.163556    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:34:16.539065 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:16.539036    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"]
Nov 02 17:34:16.539267 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:16.539236    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8" podUID="56369ef1-5944-4615-9baa-ca41c1f65b7e" containerName="limitador" containerID="cri-o://54255a4cb5d88aea6a8ad6044202385653468b907f2c6a818abb818254f23d32" gracePeriod=30
Nov 02 17:34:17.093965 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.093938    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"
Nov 02 17:34:17.190182 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.190154    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56369ef1-5944-4615-9baa-ca41c1f65b7e-config-file\" (UniqueName: \"kubernetes.io/configmap/56369ef1-5944-4615-9baa-ca41c1f65b7e-config-file\") pod \"56369ef1-5944-4615-9baa-ca41c1f65b7e\" (UID: \"56369ef1-5944-4615-9baa-ca41c1f65b7e\") "
Nov 02 17:34:17.190325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.190191    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56369ef1-5944-4615-9baa-ca41c1f65b7e-kube-api-access-gpc9l\" (UniqueName: \"kubernetes.io/projected/56369ef1-5944-4615-9baa-ca41c1f65b7e-kube-api-access-gpc9l\") pod \"56369ef1-5944-4615-9baa-ca41c1f65b7e\" (UID: \"56369ef1-5944-4615-9baa-ca41c1f65b7e\") "
Nov 02 17:34:17.190555 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.190534    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56369ef1-5944-4615-9baa-ca41c1f65b7e-config-file" pod "56369ef1-5944-4615-9baa-ca41c1f65b7e" (UID: "56369ef1-5944-4615-9baa-ca41c1f65b7e"). InnerVolumeSpecName "config-file". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:34:17.192454 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.192434    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56369ef1-5944-4615-9baa-ca41c1f65b7e-kube-api-access-gpc9l" pod "56369ef1-5944-4615-9baa-ca41c1f65b7e" (UID: "56369ef1-5944-4615-9baa-ca41c1f65b7e"). InnerVolumeSpecName "kube-api-access-gpc9l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:34:17.291248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.291219    4456 reconciler_common.go:299] "Volume detached for volume \"config-file\" (UniqueName: \"kubernetes.io/configmap/56369ef1-5944-4615-9baa-ca41c1f65b7e-config-file\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:34:17.291248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.291242    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gpc9l\" (UniqueName: \"kubernetes.io/projected/56369ef1-5944-4615-9baa-ca41c1f65b7e-kube-api-access-gpc9l\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:34:17.850979 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.850938    4456 generic.go:356] "Generic (PLEG): container finished" podID="56369ef1-5944-4615-9baa-ca41c1f65b7e" containerID="54255a4cb5d88aea6a8ad6044202385653468b907f2c6a818abb818254f23d32" exitCode=0
Nov 02 17:34:17.851158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.850984    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8" event={"ID":"56369ef1-5944-4615-9baa-ca41c1f65b7e","Type":"ContainerDied","Data":"54255a4cb5d88aea6a8ad6044202385653468b907f2c6a818abb818254f23d32"}
Nov 02 17:34:17.851158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.850992    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"
Nov 02 17:34:17.851158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.851039    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/limitador-limitador-74dfcfb894-tbfm8" event={"ID":"56369ef1-5944-4615-9baa-ca41c1f65b7e","Type":"ContainerDied","Data":"0e472ce73c453c324112c8124b45be4771351ce9994deb19b61aa2dba9db6cf3"}
Nov 02 17:34:17.851158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.851068    4456 scope.go:122] "RemoveContainer" containerID="54255a4cb5d88aea6a8ad6044202385653468b907f2c6a818abb818254f23d32"
Nov 02 17:34:17.859371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.859347    4456 scope.go:122] "RemoveContainer" containerID="54255a4cb5d88aea6a8ad6044202385653468b907f2c6a818abb818254f23d32"
Nov 02 17:34:17.859673 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:34:17.859656    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"54255a4cb5d88aea6a8ad6044202385653468b907f2c6a818abb818254f23d32\": container with ID starting with 54255a4cb5d88aea6a8ad6044202385653468b907f2c6a818abb818254f23d32 not found: ID does not exist" containerID="54255a4cb5d88aea6a8ad6044202385653468b907f2c6a818abb818254f23d32"
Nov 02 17:34:17.859714 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.859682    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"54255a4cb5d88aea6a8ad6044202385653468b907f2c6a818abb818254f23d32"} err="failed to get container status \"54255a4cb5d88aea6a8ad6044202385653468b907f2c6a818abb818254f23d32\": rpc error: code = NotFound desc = could not find container \"54255a4cb5d88aea6a8ad6044202385653468b907f2c6a818abb818254f23d32\": container with ID starting with 54255a4cb5d88aea6a8ad6044202385653468b907f2c6a818abb818254f23d32 not found: ID does not exist"
Nov 02 17:34:17.903218 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.903148    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"]
Nov 02 17:34:17.915862 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:17.915832    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/limitador-limitador-74dfcfb894-tbfm8"]
Nov 02 17:34:19.611980 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:19.611950    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56369ef1-5944-4615-9baa-ca41c1f65b7e" path="/var/lib/kubelet/pods/56369ef1-5944-4615-9baa-ca41c1f65b7e/volumes"
Nov 02 17:34:20.267376 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:20.267346    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:20.474981 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:20.474946    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:21.624065 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:21.624015    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:21.777019 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:21.776958    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:21.825631 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:21.825608    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 17:34:22.549781 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:22.549707    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:23.806259 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:23.806222    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:23.953764 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:23.953713    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:24.310178 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:24.310142    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:24.464158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:24.464123    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:24.652638 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:24.652562    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:24.814907 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:24.814874    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:32.563412 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:32.563372    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:33.225892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:33.225858    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:33.377673 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:33.377636    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:34.227386 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:34.227350    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:34.372489 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:34.372456    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:34.967444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:34.967410    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"]
Nov 02 17:34:34.967736 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:34.967724    4456 cpu_manager.go:414] "containerMap: removing container" podUID="56369ef1-5944-4615-9baa-ca41c1f65b7e" containerName="limitador"
Nov 02 17:34:34.967788 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:34.967738    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56369ef1-5944-4615-9baa-ca41c1f65b7e" containerName="limitador"
Nov 02 17:34:34.967788 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:34.967755    4456 cpu_manager.go:414] "containerMap: removing container" podUID="6fc0745b-20b4-40ee-aed8-be6323f7724f" containerName="authorino"
Nov 02 17:34:34.967788 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:34.967761    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6fc0745b-20b4-40ee-aed8-be6323f7724f" containerName="authorino"
Nov 02 17:34:34.967925 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:34.967840    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="56369ef1-5944-4615-9baa-ca41c1f65b7e" containerName="limitador"
Nov 02 17:34:34.967925 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:34.967849    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6fc0745b-20b4-40ee-aed8-be6323f7724f" containerName="authorino"
Nov 02 17:34:35.036596 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.036560    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"]
Nov 02 17:34:35.036751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.036701    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:35.071222 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.071195    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 02 17:34:35.079392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.079367    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 17:34:35.079465 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.079457    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 17:34:35.083929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.083914    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 02 17:34:35.091927 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.091915    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-bfthl\""
Nov 02 17:34:35.092017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.092004    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 02 17:34:35.148235 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.148209    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6957a218-62fc-4dd8-892e-4477b08c046f-metrics-certs\") pod \"jobset-controller-manager-77cfb5f94-gfbr9\" (UID: \"6957a218-62fc-4dd8-892e-4477b08c046f\") " pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:35.148373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.148258    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6957a218-62fc-4dd8-892e-4477b08c046f-manager-config\") pod \"jobset-controller-manager-77cfb5f94-gfbr9\" (UID: \"6957a218-62fc-4dd8-892e-4477b08c046f\") " pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:35.148432 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.148413    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6957a218-62fc-4dd8-892e-4477b08c046f-cert\") pod \"jobset-controller-manager-77cfb5f94-gfbr9\" (UID: \"6957a218-62fc-4dd8-892e-4477b08c046f\") " pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:35.148469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.148453    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hj9fz\" (UniqueName: \"kubernetes.io/projected/6957a218-62fc-4dd8-892e-4477b08c046f-kube-api-access-hj9fz\") pod \"jobset-controller-manager-77cfb5f94-gfbr9\" (UID: \"6957a218-62fc-4dd8-892e-4477b08c046f\") " pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:35.248984 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.248928    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6957a218-62fc-4dd8-892e-4477b08c046f-metrics-certs\") pod \"jobset-controller-manager-77cfb5f94-gfbr9\" (UID: \"6957a218-62fc-4dd8-892e-4477b08c046f\") " pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:35.248984 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.248962    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6957a218-62fc-4dd8-892e-4477b08c046f-manager-config\") pod \"jobset-controller-manager-77cfb5f94-gfbr9\" (UID: \"6957a218-62fc-4dd8-892e-4477b08c046f\") " pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:35.249415 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.249012    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6957a218-62fc-4dd8-892e-4477b08c046f-cert\") pod \"jobset-controller-manager-77cfb5f94-gfbr9\" (UID: \"6957a218-62fc-4dd8-892e-4477b08c046f\") " pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:35.249415 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.249037    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hj9fz\" (UniqueName: \"kubernetes.io/projected/6957a218-62fc-4dd8-892e-4477b08c046f-kube-api-access-hj9fz\") pod \"jobset-controller-manager-77cfb5f94-gfbr9\" (UID: \"6957a218-62fc-4dd8-892e-4477b08c046f\") " pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:35.249729 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.249709    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"manager-config\" (UniqueName: \"kubernetes.io/configmap/6957a218-62fc-4dd8-892e-4477b08c046f-manager-config\") pod \"jobset-controller-manager-77cfb5f94-gfbr9\" (UID: \"6957a218-62fc-4dd8-892e-4477b08c046f\") " pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:35.251548 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.251514    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cert\" (UniqueName: \"kubernetes.io/secret/6957a218-62fc-4dd8-892e-4477b08c046f-cert\") pod \"jobset-controller-manager-77cfb5f94-gfbr9\" (UID: \"6957a218-62fc-4dd8-892e-4477b08c046f\") " pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:35.251665 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.251645    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6957a218-62fc-4dd8-892e-4477b08c046f-metrics-certs\") pod \"jobset-controller-manager-77cfb5f94-gfbr9\" (UID: \"6957a218-62fc-4dd8-892e-4477b08c046f\") " pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:35.289998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.289971    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hj9fz\" (UniqueName: \"kubernetes.io/projected/6957a218-62fc-4dd8-892e-4477b08c046f-kube-api-access-hj9fz\") pod \"jobset-controller-manager-77cfb5f94-gfbr9\" (UID: \"6957a218-62fc-4dd8-892e-4477b08c046f\") " pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:35.357907 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.357840    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:35.719463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.719429    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"]
Nov 02 17:34:35.721740 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:34:35.721716    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6957a218_62fc_4dd8_892e_4477b08c046f.slice/crio-f0eee359bff3411c6ee5b38a1d43b541a1fec744900758e9c486b43a15d7491e WatchSource:0}: Error finding container f0eee359bff3411c6ee5b38a1d43b541a1fec744900758e9c486b43a15d7491e: Status 404 returned error can't find the container with id f0eee359bff3411c6ee5b38a1d43b541a1fec744900758e9c486b43a15d7491e
Nov 02 17:34:35.732629 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.732572    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:35.872713 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.872680    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:35.927506 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:35.927471    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" event={"ID":"6957a218-62fc-4dd8-892e-4477b08c046f","Type":"ContainerStarted","Data":"f0eee359bff3411c6ee5b38a1d43b541a1fec744900758e9c486b43a15d7491e"}
Nov 02 17:34:39.946182 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:39.946144    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" event={"ID":"6957a218-62fc-4dd8-892e-4477b08c046f","Type":"ContainerStarted","Data":"0cb78ab8ebb38230ce58498878fbee1c9524a897317015a5ff7bf09d770bfdec"}
Nov 02 17:34:39.946606 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:39.946252    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:39.991616 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:39.991568    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podStartSLOduration=2.7012176820000002 podStartE2EDuration="5.991552531s" totalImagesPullingTime="3.290334849s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:34:34 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:34:39.990730651 +0000 UTC m=+296.481793831" watchObservedRunningTime="2026-11-02 17:34:39.991552531 +0000 UTC m=+296.482615691"
Nov 02 17:34:42.359968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:42.359935    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:42.512289 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:42.512260    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:42.670551 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:42.670458    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:42.815718 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:42.815689    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:46.165077 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:34:46.165035    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:34:46.289019 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:46.288984    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:46.458670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:46.458635    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:46.629598 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:46.629567    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:48.056312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:48.056279    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:48.336279 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:48.336193    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:50.955969 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:50.955937    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 17:34:55.312233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:55.312196    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:55.447893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:55.447802    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:34:58.582085 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:34:58.582049    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:00.500897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:00.500859    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:00.678500 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:00.678463    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:00.848120 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:00.848042    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:00.990650 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:00.990612    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:01.380054 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:01.380020    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:03.262616 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:03.262582    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:03.411536 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:03.411502    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:03.613521 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:03.613386    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:03.775539 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:03.775507    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:16.164398 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:35:16.164358    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:35:18.365578 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:18.365522    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:18.520756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:18.520708    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:19.936977 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:19.936941    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:20.092195 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:20.092134    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:20.386470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:20.386378    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:30.332684 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:30.332650    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:30.475460 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:30.475427    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:30.895355 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:30.895322    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7984f"]
Nov 02 17:35:30.899068 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:30.899046    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7984f"
Nov 02 17:35:30.926991 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:30.926947    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kqm5s\""
Nov 02 17:35:30.927179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:30.927163    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 17:35:30.928694 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:30.928675    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7984f"]
Nov 02 17:35:30.954788 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:30.954765    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e52f91f-6424-4323-8097-03835d7ae0cf-tls-cert\") pod \"authorino-6889cbd8c4-7984f\" (UID: \"1e52f91f-6424-4323-8097-03835d7ae0cf\") " pod="kuadrant-system/authorino-6889cbd8c4-7984f"
Nov 02 17:35:30.954923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:30.954794    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6chwk\" (UniqueName: \"kubernetes.io/projected/1e52f91f-6424-4323-8097-03835d7ae0cf-kube-api-access-6chwk\") pod \"authorino-6889cbd8c4-7984f\" (UID: \"1e52f91f-6424-4323-8097-03835d7ae0cf\") " pod="kuadrant-system/authorino-6889cbd8c4-7984f"
Nov 02 17:35:31.033406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:31.033370    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7984f"]
Nov 02 17:35:31.033593 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:35:31.033576    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-6chwk tls-cert], unattached volumes=[], failed to process volumes=[]: context canceled" pod="kuadrant-system/authorino-6889cbd8c4-7984f" podUID="1e52f91f-6424-4323-8097-03835d7ae0cf"
Nov 02 17:35:31.056133 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:31.056103    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e52f91f-6424-4323-8097-03835d7ae0cf-tls-cert\") pod \"authorino-6889cbd8c4-7984f\" (UID: \"1e52f91f-6424-4323-8097-03835d7ae0cf\") " pod="kuadrant-system/authorino-6889cbd8c4-7984f"
Nov 02 17:35:31.056274 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:31.056137    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6chwk\" (UniqueName: \"kubernetes.io/projected/1e52f91f-6424-4323-8097-03835d7ae0cf-kube-api-access-6chwk\") pod \"authorino-6889cbd8c4-7984f\" (UID: \"1e52f91f-6424-4323-8097-03835d7ae0cf\") " pod="kuadrant-system/authorino-6889cbd8c4-7984f"
Nov 02 17:35:31.058647 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:31.058627    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e52f91f-6424-4323-8097-03835d7ae0cf-tls-cert\") pod \"authorino-6889cbd8c4-7984f\" (UID: \"1e52f91f-6424-4323-8097-03835d7ae0cf\") " pod="kuadrant-system/authorino-6889cbd8c4-7984f"
Nov 02 17:35:31.079624 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:31.079598    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6chwk\" (UniqueName: \"kubernetes.io/projected/1e52f91f-6424-4323-8097-03835d7ae0cf-kube-api-access-6chwk\") pod \"authorino-6889cbd8c4-7984f\" (UID: \"1e52f91f-6424-4323-8097-03835d7ae0cf\") " pod="kuadrant-system/authorino-6889cbd8c4-7984f"
Nov 02 17:35:31.174919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:31.174828    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7984f"
Nov 02 17:35:31.181887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:31.181865    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7984f"
Nov 02 17:35:31.258195 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:31.258163    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1e52f91f-6424-4323-8097-03835d7ae0cf-kube-api-access-6chwk\" (UniqueName: \"kubernetes.io/projected/1e52f91f-6424-4323-8097-03835d7ae0cf-kube-api-access-6chwk\") pod \"1e52f91f-6424-4323-8097-03835d7ae0cf\" (UID: \"1e52f91f-6424-4323-8097-03835d7ae0cf\") "
Nov 02 17:35:31.258373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:31.258265    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1e52f91f-6424-4323-8097-03835d7ae0cf-tls-cert\" (UniqueName: \"kubernetes.io/secret/1e52f91f-6424-4323-8097-03835d7ae0cf-tls-cert\") pod \"1e52f91f-6424-4323-8097-03835d7ae0cf\" (UID: \"1e52f91f-6424-4323-8097-03835d7ae0cf\") "
Nov 02 17:35:31.260369 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:31.260340    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1e52f91f-6424-4323-8097-03835d7ae0cf-kube-api-access-6chwk" pod "1e52f91f-6424-4323-8097-03835d7ae0cf" (UID: "1e52f91f-6424-4323-8097-03835d7ae0cf"). InnerVolumeSpecName "kube-api-access-6chwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:35:31.260503 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:31.260474    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1e52f91f-6424-4323-8097-03835d7ae0cf-tls-cert" pod "1e52f91f-6424-4323-8097-03835d7ae0cf" (UID: "1e52f91f-6424-4323-8097-03835d7ae0cf"). InnerVolumeSpecName "tls-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:35:31.359242 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:31.359217    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6chwk\" (UniqueName: \"kubernetes.io/projected/1e52f91f-6424-4323-8097-03835d7ae0cf-kube-api-access-6chwk\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:35:31.359242 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:31.359241    4456 reconciler_common.go:299] "Volume detached for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/1e52f91f-6424-4323-8097-03835d7ae0cf-tls-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:35:32.179308 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:32.179276    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6889cbd8c4-7984f"
Nov 02 17:35:32.263701 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:32.263672    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7984f"]
Nov 02 17:35:32.267612 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:32.267583    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/authorino-6889cbd8c4-7984f"]
Nov 02 17:35:32.684801 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:32.684765    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:35:33.612299 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:35:33.612268    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1e52f91f-6424-4323-8097-03835d7ae0cf" path="/var/lib/kubelet/pods/1e52f91f-6424-4323-8097-03835d7ae0cf/volumes"
Nov 02 17:35:46.164488 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:35:46.164450    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:36:16.164268 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:36:16.164208    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:36:28.339899 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:28.339798    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:28.500831 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:28.500778    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:32.812373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:32.812336    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:46.163803 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:36:46.163763    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:36:48.219681 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:48.219647    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:48.758338 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:48.758300    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:48.902982 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:48.902950    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:49.911469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:49.911439    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"]
Nov 02 17:36:49.915151 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:49.915133    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:49.934145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:49.934116    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"]
Nov 02 17:36:49.944524 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:49.944501    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 17:36:49.944760 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:49.944740    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 02 17:36:49.944932 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:49.944917    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 17:36:49.944986 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:49.944919    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f7th6\""
Nov 02 17:36:49.995671 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:49.995638    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:50.058107 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.058073    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.058298 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.058127    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpdns\" (UniqueName: \"kubernetes.io/projected/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-kube-api-access-lpdns\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.058298 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.058196    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.058298 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.058228    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.058298 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.058247    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.058475 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.058391    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.159528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.159489    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.159755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.159543    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpdns\" (UniqueName: \"kubernetes.io/projected/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-kube-api-access-lpdns\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.159755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.159564    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.159755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.159585    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.159755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.159708    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.160038 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.159789    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.160103 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.160037    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-model-cache\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.160103 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.160067    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-home\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.160207 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.160139    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-tmp-dir\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.162033 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.161981    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-dshm\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.162263 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.162237    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-tls-certs\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.184017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.183994    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpdns\" (UniqueName: \"kubernetes.io/projected/3b0e614e-16b1-4e28-bea6-4b7b9a62b177-kube-api-access-lpdns\") pod \"e2e-trlp-test-simulated-kserve-69b588897d-xc9kg\" (UID: \"3b0e614e-16b1-4e28-bea6-4b7b9a62b177\") " pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.229608 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.229572    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:50.530808 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.530782    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"]
Nov 02 17:36:50.533458 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:36:50.533431    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3b0e614e_16b1_4e28_bea6_4b7b9a62b177.slice/crio-a064fab5183270131c349f6535c7f405948d6ad7e2057592fe5e5539d743caf5 WatchSource:0}: Error finding container a064fab5183270131c349f6535c7f405948d6ad7e2057592fe5e5539d743caf5: Status 404 returned error can't find the container with id a064fab5183270131c349f6535c7f405948d6ad7e2057592fe5e5539d743caf5
Nov 02 17:36:50.534850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:50.534832    4456 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:36:51.510730 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:51.510683    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" event={"ID":"3b0e614e-16b1-4e28-bea6-4b7b9a62b177","Type":"ContainerStarted","Data":"a064fab5183270131c349f6535c7f405948d6ad7e2057592fe5e5539d743caf5"}
Nov 02 17:36:52.453356 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:52.453311    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:52.517017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:52.516977    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" event={"ID":"3b0e614e-16b1-4e28-bea6-4b7b9a62b177","Type":"ContainerStarted","Data":"b95001690ccae1b39e5a69ee364d7aa2ca7000284ffad2460761d0f764896501"}
Nov 02 17:36:52.561560 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:52.561499    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podStartSLOduration=2.462835326 podStartE2EDuration="3.561485111s" totalImagesPullingTime="1.098649785s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:52.560173504 +0000 UTC m=+429.051236675" watchObservedRunningTime="2026-11-02 17:36:52.561485111 +0000 UTC m=+429.052548290"
Nov 02 17:36:53.230153 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:53.230118    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:36:54.406125 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:54.405925    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:55.879960 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:55.879925    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"]
Nov 02 17:36:55.883664 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:55.883643    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:55.911743 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:55.911712    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 02 17:36:55.914449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:55.914424    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"]
Nov 02 17:36:55.977524 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:55.977490    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:56.010736 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.010708    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.010736 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.010739    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.010920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.010782    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.010920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.010879    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.010994 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.010921    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.010994 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.010945    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv8xf\" (UniqueName: \"kubernetes.io/projected/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-kube-api-access-gv8xf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.111547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.111514    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.111547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.111550    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.111773 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.111591    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.111773 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.111621    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.111773 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.111650    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.111773 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.111670    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv8xf\" (UniqueName: \"kubernetes.io/projected/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-kube-api-access-gv8xf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.112104 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.112074    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-home\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.112200 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.112132    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-tmp-dir\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.112200 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.112189    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-model-cache\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.114206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.114184    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-dshm\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.114557 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.114539    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-tls-certs\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.135854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.135781    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv8xf\" (UniqueName: \"kubernetes.io/projected/14d43510-cfbb-4ddc-88cb-ccd4cb3a543d-kube-api-access-gv8xf\") pod \"facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k\" (UID: \"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d\") " pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.199461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.199426    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:56.511183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.511023    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"]
Nov 02 17:36:56.513668 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:36:56.513640    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod14d43510_cfbb_4ddc_88cb_ccd4cb3a543d.slice/crio-ccc5720a6b8363afeeff8a55d65f686d9e34d9227502031b8f8d763fb908a2f3 WatchSource:0}: Error finding container ccc5720a6b8363afeeff8a55d65f686d9e34d9227502031b8f8d763fb908a2f3: Status 404 returned error can't find the container with id ccc5720a6b8363afeeff8a55d65f686d9e34d9227502031b8f8d763fb908a2f3
Nov 02 17:36:56.534769 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:56.534738    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" event={"ID":"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d","Type":"ContainerStarted","Data":"ccc5720a6b8363afeeff8a55d65f686d9e34d9227502031b8f8d763fb908a2f3"}
Nov 02 17:36:57.479379 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:57.479340    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:57.540469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:57.540434    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" event={"ID":"14d43510-cfbb-4ddc-88cb-ccd4cb3a543d","Type":"ContainerStarted","Data":"b748a1c2f46a109eb72043e1448142b0a3ef7d0436159151465108bd48b99ed5"}
Nov 02 17:36:57.613842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:57.613782    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podStartSLOduration=2.403641807 podStartE2EDuration="2.613767627s" totalImagesPullingTime="210.12582ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:55 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:36:57.612140973 +0000 UTC m=+434.103204146" watchObservedRunningTime="2026-11-02 17:36:57.613767627 +0000 UTC m=+434.104830766"
Nov 02 17:36:58.199773 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:58.199725    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:36:59.621247 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.621218    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"]
Nov 02 17:36:59.624786 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.624764    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.641520 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.641499    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 02 17:36:59.644139 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.644119    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e44698a-838e-4eea-b956-585673963ed9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.644139 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.644148    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e44698a-838e-4eea-b956-585673963ed9-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.644312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.644192    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e44698a-838e-4eea-b956-585673963ed9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.644312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.644208    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f4rr6\" (UniqueName: \"kubernetes.io/projected/6e44698a-838e-4eea-b956-585673963ed9-kube-api-access-f4rr6\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.644312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.644274    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e44698a-838e-4eea-b956-585673963ed9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.644472 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.644354    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e44698a-838e-4eea-b956-585673963ed9-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.657024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.657000    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"]
Nov 02 17:36:59.723049 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.723013    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:36:59.745190 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.745156    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e44698a-838e-4eea-b956-585673963ed9-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.745330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.745206    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e44698a-838e-4eea-b956-585673963ed9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.745373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.745328    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e44698a-838e-4eea-b956-585673963ed9-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.745373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.745362    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e44698a-838e-4eea-b956-585673963ed9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.745481 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.745388    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f4rr6\" (UniqueName: \"kubernetes.io/projected/6e44698a-838e-4eea-b956-585673963ed9-kube-api-access-f4rr6\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.745481 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.745452    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e44698a-838e-4eea-b956-585673963ed9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.745612 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.745591    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/6e44698a-838e-4eea-b956-585673963ed9-home\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.745781 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.745750    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/6e44698a-838e-4eea-b956-585673963ed9-tmp-dir\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.745781 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.745767    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/6e44698a-838e-4eea-b956-585673963ed9-model-cache\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.747671 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.747647    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/6e44698a-838e-4eea-b956-585673963ed9-dshm\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.747953 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.747936    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/6e44698a-838e-4eea-b956-585673963ed9-tls-certs\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.769848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.769794    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f4rr6\" (UniqueName: \"kubernetes.io/projected/6e44698a-838e-4eea-b956-585673963ed9-kube-api-access-f4rr6\") pod \"e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4\" (UID: \"6e44698a-838e-4eea-b956-585673963ed9\") " pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:36:59.939700 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:36:59.939664    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:37:00.230860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:00.230760    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:37:00.245705 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:00.245671    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:37:00.245907 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:00.245886    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"]
Nov 02 17:37:00.249348 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:37:00.249322    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6e44698a_838e_4eea_b956_585673963ed9.slice/crio-394c88b54416986419439d500acf2de036f6aa42bb31c83fa851eacda1a2aae6 WatchSource:0}: Error finding container 394c88b54416986419439d500acf2de036f6aa42bb31c83fa851eacda1a2aae6: Status 404 returned error can't find the container with id 394c88b54416986419439d500acf2de036f6aa42bb31c83fa851eacda1a2aae6
Nov 02 17:37:00.554152 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:00.554115    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" event={"ID":"6e44698a-838e-4eea-b956-585673963ed9","Type":"ContainerStarted","Data":"7e68c2b39ae4cecf707b2ab53563bed47fa8494473e6dfa591c94dd630cdbabc"}
Nov 02 17:37:00.554320 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:00.554178    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" event={"ID":"6e44698a-838e-4eea-b956-585673963ed9","Type":"ContainerStarted","Data":"394c88b54416986419439d500acf2de036f6aa42bb31c83fa851eacda1a2aae6"}
Nov 02 17:37:00.565745 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:00.565720    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 17:37:00.598160 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:00.598108    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podStartSLOduration=1.375688537 podStartE2EDuration="1.59809552s" totalImagesPullingTime="222.406983ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:36:59 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:00.59713267 +0000 UTC m=+437.088195851" watchObservedRunningTime="2026-11-02 17:37:00.59809552 +0000 UTC m=+437.089158680"
Nov 02 17:37:00.940662 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:00.940633    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:37:01.971511 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:01.971471    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:06.200107 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:06.200070    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:37:06.214164 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:06.214138    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:37:06.591929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:06.591846    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 17:37:09.940465 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:09.940427    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:37:09.952693 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:09.952668    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:37:10.614996 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:10.614970    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 17:37:13.917411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:13.917376    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/authorino-6f5b7cd5c7-brdvs"]
Nov 02 17:37:13.921051 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:13.921031    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f5b7cd5c7-brdvs"
Nov 02 17:37:13.940234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:13.940208    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f5b7cd5c7-brdvs"]
Nov 02 17:37:13.950788 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:13.950764    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 17:37:13.950979 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:13.950959    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kqm5s\""
Nov 02 17:37:13.970177 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:13.970155    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rbcsx\" (UniqueName: \"kubernetes.io/projected/040ec9bf-319e-4bae-b233-4e4e7e74a379-kube-api-access-rbcsx\") pod \"authorino-6f5b7cd5c7-brdvs\" (UID: \"040ec9bf-319e-4bae-b233-4e4e7e74a379\") " pod="kuadrant-system/authorino-6f5b7cd5c7-brdvs"
Nov 02 17:37:13.970316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:13.970216    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/040ec9bf-319e-4bae-b233-4e4e7e74a379-tls-cert\") pod \"authorino-6f5b7cd5c7-brdvs\" (UID: \"040ec9bf-319e-4bae-b233-4e4e7e74a379\") " pod="kuadrant-system/authorino-6f5b7cd5c7-brdvs"
Nov 02 17:37:14.070775 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:14.070735    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rbcsx\" (UniqueName: \"kubernetes.io/projected/040ec9bf-319e-4bae-b233-4e4e7e74a379-kube-api-access-rbcsx\") pod \"authorino-6f5b7cd5c7-brdvs\" (UID: \"040ec9bf-319e-4bae-b233-4e4e7e74a379\") " pod="kuadrant-system/authorino-6f5b7cd5c7-brdvs"
Nov 02 17:37:14.070990 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:14.070856    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/040ec9bf-319e-4bae-b233-4e4e7e74a379-tls-cert\") pod \"authorino-6f5b7cd5c7-brdvs\" (UID: \"040ec9bf-319e-4bae-b233-4e4e7e74a379\") " pod="kuadrant-system/authorino-6f5b7cd5c7-brdvs"
Nov 02 17:37:14.073455 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:14.073431    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-cert\" (UniqueName: \"kubernetes.io/secret/040ec9bf-319e-4bae-b233-4e4e7e74a379-tls-cert\") pod \"authorino-6f5b7cd5c7-brdvs\" (UID: \"040ec9bf-319e-4bae-b233-4e4e7e74a379\") " pod="kuadrant-system/authorino-6f5b7cd5c7-brdvs"
Nov 02 17:37:14.097278 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:14.097256    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rbcsx\" (UniqueName: \"kubernetes.io/projected/040ec9bf-319e-4bae-b233-4e4e7e74a379-kube-api-access-rbcsx\") pod \"authorino-6f5b7cd5c7-brdvs\" (UID: \"040ec9bf-319e-4bae-b233-4e4e7e74a379\") " pod="kuadrant-system/authorino-6f5b7cd5c7-brdvs"
Nov 02 17:37:14.233287 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:14.233253    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/authorino-6f5b7cd5c7-brdvs"
Nov 02 17:37:14.535454 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:14.535425    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/authorino-6f5b7cd5c7-brdvs"]
Nov 02 17:37:14.538123 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:37:14.538094    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod040ec9bf_319e_4bae_b233_4e4e7e74a379.slice/crio-f9e781dec927babf153f52d47d4e502fdcb99cfd1e49569318b5f802a6bc70cd WatchSource:0}: Error finding container f9e781dec927babf153f52d47d4e502fdcb99cfd1e49569318b5f802a6bc70cd: Status 404 returned error can't find the container with id f9e781dec927babf153f52d47d4e502fdcb99cfd1e49569318b5f802a6bc70cd
Nov 02 17:37:14.618996 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:14.618962    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f5b7cd5c7-brdvs" event={"ID":"040ec9bf-319e-4bae-b233-4e4e7e74a379","Type":"ContainerStarted","Data":"f9e781dec927babf153f52d47d4e502fdcb99cfd1e49569318b5f802a6bc70cd"}
Nov 02 17:37:15.626176 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:15.626081    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/authorino-6f5b7cd5c7-brdvs" event={"ID":"040ec9bf-319e-4bae-b233-4e4e7e74a379","Type":"ContainerStarted","Data":"cb8c1cf326798f185a636eb123063550934053d27d3ad606650f854f6d9fb516"}
Nov 02 17:37:15.667846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:15.667777    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/authorino-6f5b7cd5c7-brdvs" podStartSLOduration=1.95675554 podStartE2EDuration="2.667763312s" totalImagesPullingTime="711.007772ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:13 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:15.666505209 +0000 UTC m=+452.157568370" watchObservedRunningTime="2026-11-02 17:37:15.667763312 +0000 UTC m=+452.158826474"
Nov 02 17:37:16.164630 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:37:16.164592    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:37:16.252453 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.252410    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"]
Nov 02 17:37:16.256301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.256276    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"
Nov 02 17:37:16.289248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.289220    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-dockercfg-rwt92\""
Nov 02 17:37:16.289517 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.289494    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub\"/\"ai-gateway-controller-metrics-tls\""
Nov 02 17:37:16.289629 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.289528    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"kube-root-ca.crt\""
Nov 02 17:37:16.289629 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.289529    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"openshift-service-ca.crt\""
Nov 02 17:37:16.289750 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.289732    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub\"/\"ai-gateway-controller-parameters\""
Nov 02 17:37:16.292395 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.292373    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"]
Nov 02 17:37:16.299584 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.299562    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lklcf\" (UniqueName: \"kubernetes.io/projected/a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7-kube-api-access-lklcf\") pod \"ai-gateway-controller-7cb8f6bb64-5ptj6\" (UID: \"a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"
Nov 02 17:37:16.299691 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.299600    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-5ptj6\" (UID: \"a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"
Nov 02 17:37:16.300128 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.300107    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:16.400395 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.400354    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lklcf\" (UniqueName: \"kubernetes.io/projected/a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7-kube-api-access-lklcf\") pod \"ai-gateway-controller-7cb8f6bb64-5ptj6\" (UID: \"a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"
Nov 02 17:37:16.400395 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.400399    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-5ptj6\" (UID: \"a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"
Nov 02 17:37:16.403360 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.403333    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7-metrics-certs\") pod \"ai-gateway-controller-7cb8f6bb64-5ptj6\" (UID: \"a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"
Nov 02 17:37:16.432888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.432455    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lklcf\" (UniqueName: \"kubernetes.io/projected/a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7-kube-api-access-lklcf\") pod \"ai-gateway-controller-7cb8f6bb64-5ptj6\" (UID: \"a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7\") " pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"
Nov 02 17:37:16.458355 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.458317    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:16.568866 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.568834    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"
Nov 02 17:37:16.901042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:16.900626    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"]
Nov 02 17:37:16.903785 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:37:16.903749    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6d4b4e4_a41a_4e71_854e_8d7cc8a39ef7.slice/crio-6c16c321f524e95ab704043faf5356de6ecbccf1d42bf62963ab699e7c011a96 WatchSource:0}: Error finding container 6c16c321f524e95ab704043faf5356de6ecbccf1d42bf62963ab699e7c011a96: Status 404 returned error can't find the container with id 6c16c321f524e95ab704043faf5356de6ecbccf1d42bf62963ab699e7c011a96
Nov 02 17:37:17.641101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:17.641058    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6" event={"ID":"a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7","Type":"ContainerStarted","Data":"6c16c321f524e95ab704043faf5356de6ecbccf1d42bf62963ab699e7c011a96"}
Nov 02 17:37:20.661364 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:20.661326    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6" event={"ID":"a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7","Type":"ContainerStarted","Data":"f182be7af8116b58c05d099aa07acc3cf2c761fa2d83e0d97ae4c93ffb644c63"}
Nov 02 17:37:20.661799 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:20.661417    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"
Nov 02 17:37:20.708706 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:20.708634    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6" podStartSLOduration=1.87811546 podStartE2EDuration="4.708615994s" totalImagesPullingTime="2.830500534s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:16 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:20.707508963 +0000 UTC m=+457.198572126" watchObservedRunningTime="2026-11-02 17:37:20.708615994 +0000 UTC m=+457.199679156"
Nov 02 17:37:22.177912 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:22.177882    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:22.323626 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:22.323594    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:31.669145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:31.669113    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"
Nov 02 17:37:32.775351 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.775311    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"]
Nov 02 17:37:32.779138 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.779115    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.799444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.799421    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"]
Nov 02 17:37:32.802858 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.802839    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.810273 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.810257    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 17:37:32.810366 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.810304    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lkf8q\""
Nov 02 17:37:32.812604 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.812582    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"]
Nov 02 17:37:32.837728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.837701    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"]
Nov 02 17:37:32.855096 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855069    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.855228 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855106    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.855228 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855135    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.855319 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855218    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.855319 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855256    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.855319 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855298    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a245c75-8896-4899-9353-7460adf4550a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.855437 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855372    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.855437 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855391    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.855545 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855447    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mxlhh\" (UniqueName: \"kubernetes.io/projected/5a245c75-8896-4899-9353-7460adf4550a-kube-api-access-mxlhh\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.855545 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855483    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a245c75-8896-4899-9353-7460adf4550a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.855655 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855545    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.855655 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855585    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a245c75-8896-4899-9353-7460adf4550a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.855655 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855611    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a66a57d-f1df-48f1-8715-92a9dab50057-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.855655 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855641    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.855804 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855705    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-brj4f\" (UniqueName: \"kubernetes.io/projected/1a66a57d-f1df-48f1-8715-92a9dab50057-kube-api-access-brj4f\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.855804 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855748    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.855804 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855768    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.855949 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.855842    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.956435 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.956396    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a245c75-8896-4899-9353-7460adf4550a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.956435 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.956442    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.956684 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.956463    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.956684 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.956485    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mxlhh\" (UniqueName: \"kubernetes.io/projected/5a245c75-8896-4899-9353-7460adf4550a-kube-api-access-mxlhh\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.956684 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.956506    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a245c75-8896-4899-9353-7460adf4550a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.956684 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.956542    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.956684 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.956571    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a245c75-8896-4899-9353-7460adf4550a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.956684 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.956597    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a66a57d-f1df-48f1-8715-92a9dab50057-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.957029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.956888    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.957029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.956941    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-brj4f\" (UniqueName: \"kubernetes.io/projected/1a66a57d-f1df-48f1-8715-92a9dab50057-kube-api-access-brj4f\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.957029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.956948    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.957029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.956998    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.957400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.957035    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.957400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.957083    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.957400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.957121    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.957400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.957146    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.957400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.957174    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.957400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.957204    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.957400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.957232    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.957400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.957277    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a245c75-8896-4899-9353-7460adf4550a-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.957400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.957304    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.957400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.957327    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a66a57d-f1df-48f1-8715-92a9dab50057-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.957959 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.957471    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.957959 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.957538    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.957959 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.957587    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.957959 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.957922    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.958171 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.958054    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.958171 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.958127    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.959206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.959181    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.959862 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.959836    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:32.959862 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.959847    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.959995 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.959965    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a245c75-8896-4899-9353-7460adf4550a-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:32.994106 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:32.994074    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:33.000595 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:33.000563    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a245c75-8896-4899-9353-7460adf4550a-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:33.000756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:33.000732    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:33.000941 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:33.000923    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-brj4f\" (UniqueName: \"kubernetes.io/projected/1a66a57d-f1df-48f1-8715-92a9dab50057-kube-api-access-brj4f\") pod \"maas-default-gateway-openshift-default-76f4876667-c5fdk\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:33.001328 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:33.001310    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mxlhh\" (UniqueName: \"kubernetes.io/projected/5a245c75-8896-4899-9353-7460adf4550a-kube-api-access-mxlhh\") pod \"maas-default-gateway-openshift-default-76f4876667-vlw54\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:33.093456 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:33.093375    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:33.117980 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:33.117952    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:33.436288 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:33.436258    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"]
Nov 02 17:37:33.437770 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:33.437741    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"]
Nov 02 17:37:33.438155 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:37:33.438127    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1a66a57d_f1df_48f1_8715_92a9dab50057.slice/crio-b9fd2a4e22caf4eaa8cb6415e5aab7a97b4de54c14abfd143539dd5aa44179b8 WatchSource:0}: Error finding container b9fd2a4e22caf4eaa8cb6415e5aab7a97b4de54c14abfd143539dd5aa44179b8: Status 404 returned error can't find the container with id b9fd2a4e22caf4eaa8cb6415e5aab7a97b4de54c14abfd143539dd5aa44179b8
Nov 02 17:37:33.439246 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:37:33.439224    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5a245c75_8896_4899_9353_7460adf4550a.slice/crio-f760f0085515067c3b0f4fce03bb3e1caddc6d00943d5ccd35e9472c8d1ee91d WatchSource:0}: Error finding container f760f0085515067c3b0f4fce03bb3e1caddc6d00943d5ccd35e9472c8d1ee91d: Status 404 returned error can't find the container with id f760f0085515067c3b0f4fce03bb3e1caddc6d00943d5ccd35e9472c8d1ee91d
Nov 02 17:37:33.719279 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:33.719199    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54" event={"ID":"5a245c75-8896-4899-9353-7460adf4550a","Type":"ContainerStarted","Data":"f760f0085515067c3b0f4fce03bb3e1caddc6d00943d5ccd35e9472c8d1ee91d"}
Nov 02 17:37:33.720438 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:33.720413    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk" event={"ID":"1a66a57d-f1df-48f1-8715-92a9dab50057","Type":"ContainerStarted","Data":"b9fd2a4e22caf4eaa8cb6415e5aab7a97b4de54c14abfd143539dd5aa44179b8"}
Nov 02 17:37:36.257219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:36.257182    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:37:36.257629 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:36.257268    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:37:36.257629 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:36.257315    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:37:36.351058 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:36.351015    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:37:36.351168 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:36.351149    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:37:36.351225 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:36.351209    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:37:36.738850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:36.738781    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54" event={"ID":"5a245c75-8896-4899-9353-7460adf4550a","Type":"ContainerStarted","Data":"d087b4bc5bcd54eed19435a49f0b6879f565e02ac4f7996dee47d18d3c46d394"}
Nov 02 17:37:36.740852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:36.740794    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk" event={"ID":"1a66a57d-f1df-48f1-8715-92a9dab50057","Type":"ContainerStarted","Data":"b78faf2746e6fa586a94ea96d8e2f14c41ba5043250d3ae9608f82fdb6426bbc"}
Nov 02 17:37:36.836881 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:36.836530    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54" podStartSLOduration=2.021205168 podStartE2EDuration="4.836515092s" totalImagesPullingTime="2.815309924s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:36.823298179 +0000 UTC m=+473.314361363" watchObservedRunningTime="2026-11-02 17:37:36.836515092 +0000 UTC m=+473.327578253"
Nov 02 17:37:36.871754 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:36.871698    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk" podStartSLOduration=1.961988791 podStartE2EDuration="4.871684378s" totalImagesPullingTime="2.909695587s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:32 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:36.869703848 +0000 UTC m=+473.360767022" watchObservedRunningTime="2026-11-02 17:37:36.871684378 +0000 UTC m=+473.362747538"
Nov 02 17:37:37.099894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:37.098984    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:37.107840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:37.107007    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vlw54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 02 17:37:37.107840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:37.107076    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54" podUID="5a245c75-8896-4899-9353-7460adf4550a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 02 17:37:37.130841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:37.121889    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:37.130841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:37.127943    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c5fdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 17:37:37.130841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:37.128003    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk" podUID="1a66a57d-f1df-48f1-8715-92a9dab50057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 17:37:38.101842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:38.101291    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vlw54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 02 17:37:38.101842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:38.101350    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54" podUID="5a245c75-8896-4899-9353-7460adf4550a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 02 17:37:38.119267 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:38.119152    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c5fdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 17:37:38.119267 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:38.119210    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk" podUID="1a66a57d-f1df-48f1-8715-92a9dab50057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 17:37:39.097840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:39.097225    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vlw54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 02 17:37:39.097840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:39.097299    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54" podUID="5a245c75-8896-4899-9353-7460adf4550a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 02 17:37:39.119261 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:39.119227    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c5fdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 17:37:39.119886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:39.119855    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk" podUID="1a66a57d-f1df-48f1-8715-92a9dab50057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 17:37:40.099842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:40.099237    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vlw54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 02 17:37:40.099842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:40.099315    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54" podUID="5a245c75-8896-4899-9353-7460adf4550a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 02 17:37:40.122839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:40.122201    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c5fdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 17:37:40.122839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:40.122264    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk" podUID="1a66a57d-f1df-48f1-8715-92a9dab50057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 17:37:40.273283 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:40.273237    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:41.097304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:41.097261    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vlw54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 02 17:37:41.097601 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:41.097571    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54" podUID="5a245c75-8896-4899-9353-7460adf4550a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 02 17:37:41.119838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:41.119774    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c5fdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 17:37:41.120037 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:41.119868    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk" podUID="1a66a57d-f1df-48f1-8715-92a9dab50057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 17:37:42.104851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:42.097216    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vlw54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 02 17:37:42.104851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:42.097279    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54" podUID="5a245c75-8896-4899-9353-7460adf4550a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 02 17:37:42.121564 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:42.121529    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c5fdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 17:37:42.121857 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:42.121799    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk" podUID="1a66a57d-f1df-48f1-8715-92a9dab50057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 17:37:43.094415 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:43.094379    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vlw54 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused" start-of-body=
Nov 02 17:37:43.094590 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:43.094429    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54" podUID="5a245c75-8896-4899-9353-7460adf4550a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.35:15021/healthz/ready\": dial tcp 10.134.0.35:15021: connect: connection refused"
Nov 02 17:37:43.119161 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:43.119125    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-c5fdk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused" start-of-body=
Nov 02 17:37:43.119477 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:43.119193    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk" podUID="1a66a57d-f1df-48f1-8715-92a9dab50057" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.36:15021/healthz/ready\": dial tcp 10.134.0.36:15021: connect: connection refused"
Nov 02 17:37:44.596224 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:44.596191    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:44.597293 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:44.597272    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:44.597996 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:44.597978    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:37:44.599119 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:44.599100    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:44.599639 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:44.599618    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:44.600488 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:44.600469    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:37:44.885397 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:44.885312    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:45.131199 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.131165    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:45.436756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.436721    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"]
Nov 02 17:37:45.436989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.436965    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6" podUID="a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7" containerName="manager" containerID="cri-o://f182be7af8116b58c05d099aa07acc3cf2c761fa2d83e0d97ae4c93ffb644c63" gracePeriod=30
Nov 02 17:37:45.687988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.687915    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"
Nov 02 17:37:45.715469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.715440    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7-metrics-certs\") pod \"a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7\" (UID: \"a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7\") "
Nov 02 17:37:45.715604 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.715478    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7-kube-api-access-lklcf\" (UniqueName: \"kubernetes.io/projected/a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7-kube-api-access-lklcf\") pod \"a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7\" (UID: \"a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7\") "
Nov 02 17:37:45.717854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.717807    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7-metrics-certs" pod "a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7" (UID: "a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:37:45.717949 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.717890    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7-kube-api-access-lklcf" pod "a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7" (UID: "a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7"). InnerVolumeSpecName "kube-api-access-lklcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:37:45.783734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.783703    4456 generic.go:356] "Generic (PLEG): container finished" podID="a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7" containerID="f182be7af8116b58c05d099aa07acc3cf2c761fa2d83e0d97ae4c93ffb644c63" exitCode=0
Nov 02 17:37:45.783921 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.783757    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"
Nov 02 17:37:45.783921 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.783802    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6" event={"ID":"a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7","Type":"ContainerDied","Data":"f182be7af8116b58c05d099aa07acc3cf2c761fa2d83e0d97ae4c93ffb644c63"}
Nov 02 17:37:45.783921 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.783880    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6" event={"ID":"a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7","Type":"ContainerDied","Data":"6c16c321f524e95ab704043faf5356de6ecbccf1d42bf62963ab699e7c011a96"}
Nov 02 17:37:45.783921 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.783911    4456 scope.go:122] "RemoveContainer" containerID="f182be7af8116b58c05d099aa07acc3cf2c761fa2d83e0d97ae4c93ffb644c63"
Nov 02 17:37:45.793513 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.793498    4456 scope.go:122] "RemoveContainer" containerID="f182be7af8116b58c05d099aa07acc3cf2c761fa2d83e0d97ae4c93ffb644c63"
Nov 02 17:37:45.793770 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:37:45.793752    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f182be7af8116b58c05d099aa07acc3cf2c761fa2d83e0d97ae4c93ffb644c63\": container with ID starting with f182be7af8116b58c05d099aa07acc3cf2c761fa2d83e0d97ae4c93ffb644c63 not found: ID does not exist" containerID="f182be7af8116b58c05d099aa07acc3cf2c761fa2d83e0d97ae4c93ffb644c63"
Nov 02 17:37:45.793854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.793778    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f182be7af8116b58c05d099aa07acc3cf2c761fa2d83e0d97ae4c93ffb644c63"} err="failed to get container status \"f182be7af8116b58c05d099aa07acc3cf2c761fa2d83e0d97ae4c93ffb644c63\": rpc error: code = NotFound desc = could not find container \"f182be7af8116b58c05d099aa07acc3cf2c761fa2d83e0d97ae4c93ffb644c63\": container with ID starting with f182be7af8116b58c05d099aa07acc3cf2c761fa2d83e0d97ae4c93ffb644c63 not found: ID does not exist"
Nov 02 17:37:45.817323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.817304    4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7-metrics-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:37:45.817323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.817328    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lklcf\" (UniqueName: \"kubernetes.io/projected/a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7-kube-api-access-lklcf\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:37:45.834515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.834489    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"]
Nov 02 17:37:45.839231 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:45.839165    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["opendatahub/ai-gateway-controller-7cb8f6bb64-5ptj6"]
Nov 02 17:37:46.023686 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:46.023586    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:46.164273 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:37:46.164241    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:37:47.316756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.316723    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:47.613029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.612951    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7" path="/var/lib/kubelet/pods/a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7/volumes"
Nov 02 17:37:47.845538 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.845504    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"]
Nov 02 17:37:47.845927 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.845913    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7" containerName="manager"
Nov 02 17:37:47.845996 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.845929    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7" containerName="manager"
Nov 02 17:37:47.845996 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.845988    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6d4b4e4-a41a-4e71-854e-8d7cc8a39ef7" containerName="manager"
Nov 02 17:37:47.850389 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.850369    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:47.871541 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.871479    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"]
Nov 02 17:37:47.928206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.928170    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:47.937407 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.937383    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:47.937524 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.937425    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:47.937524 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.937502    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:47.937607 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.937530    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:47.937607 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.937550    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:47.937674 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.937642    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:47.937718 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.937673    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280a3e6c-c540-4c66-b963-c678ddca6e40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:47.937764 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.937744    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:47.937806 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:47.937778    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5f64s\" (UniqueName: \"kubernetes.io/projected/280a3e6c-c540-4c66-b963-c678ddca6e40-kube-api-access-5f64s\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.039098 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.039058    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280a3e6c-c540-4c66-b963-c678ddca6e40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.039278 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.039105    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.039278 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.039136    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5f64s\" (UniqueName: \"kubernetes.io/projected/280a3e6c-c540-4c66-b963-c678ddca6e40-kube-api-access-5f64s\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.039278 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.039177    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.039278 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.039213    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.039510 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.039340    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.039510 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.039397    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.039510 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.039430    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.039510 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.039495    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.039715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.039555    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.039715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.039596    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.039840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.039741    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.039903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.039847    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280a3e6c-c540-4c66-b963-c678ddca6e40-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.039903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.039872    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.041906 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.041880    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.041995 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.041975    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.063046 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.063024    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.063189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.063163    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5f64s\" (UniqueName: \"kubernetes.io/projected/280a3e6c-c540-4c66-b963-c678ddca6e40-kube-api-access-5f64s\") pod \"maas-default-gateway-openshift-default-76f4876667-5bqck\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.165051 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.164958    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:48.485542 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.485417    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"]
Nov 02 17:37:48.487913 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:37:48.487885    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod280a3e6c_c540_4c66_b963_c678ddca6e40.slice/crio-3d9b8adee15845131a45f38a576a6f1283f6e2b9826d6dac59798f22488855ec WatchSource:0}: Error finding container 3d9b8adee15845131a45f38a576a6f1283f6e2b9826d6dac59798f22488855ec: Status 404 returned error can't find the container with id 3d9b8adee15845131a45f38a576a6f1283f6e2b9826d6dac59798f22488855ec
Nov 02 17:37:48.490129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.490100    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:37:48.490211 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.490160    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:37:48.490211 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.490188    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:37:48.806006 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.805921    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" event={"ID":"280a3e6c-c540-4c66-b963-c678ddca6e40","Type":"ContainerStarted","Data":"60addef8887b57cc6daa92f4b0dbb1eb9ed3aa5f9e9c2ecde7a1e071117ccff8"}
Nov 02 17:37:48.806006 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.805998    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" event={"ID":"280a3e6c-c540-4c66-b963-c678ddca6e40","Type":"ContainerStarted","Data":"3d9b8adee15845131a45f38a576a6f1283f6e2b9826d6dac59798f22488855ec"}
Nov 02 17:37:48.867918 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:48.867867    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" podStartSLOduration=1.867853529 podStartE2EDuration="1.867853529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:37:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:37:48.865574438 +0000 UTC m=+485.356637601" watchObservedRunningTime="2026-11-02 17:37:48.867853529 +0000 UTC m=+485.358916686"
Nov 02 17:37:49.165676 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:49.165587    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:49.167155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:49.167127    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5bqck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 17:37:49.167281 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:49.167190    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 17:37:49.679187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:49.679130    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:49.819527 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:49.819469    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:50.165267 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:50.165237    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5bqck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 17:37:50.165431 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:50.165286    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 17:37:51.166131 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:51.166097    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5bqck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 17:37:51.166555 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:51.166147    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 17:37:51.317493 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:51.317455    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:51.503469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:51.503359    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:37:52.165566 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:52.165535    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5bqck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 17:37:52.165747 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:52.165586    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 17:37:53.165680 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:53.165637    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5bqck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 17:37:53.166115 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:53.165696    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 17:37:54.165743 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:54.165706    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5bqck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 17:37:54.166199 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:54.165764    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 17:37:55.165702 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:55.165662    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5bqck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused" start-of-body=
Nov 02 17:37:55.165956 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:55.165726    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": dial tcp 10.134.0.37:15021: connect: connection refused"
Nov 02 17:37:56.689856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:56.689700    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:56.690320 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:56.690188    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:37:56.690919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:37:56.690900    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 17:38:14.563299 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:14.563264    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:14.755838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:14.738945    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:15.016693 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.016665    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"]
Nov 02 17:38:15.023162 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.023141    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.045436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.045409    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"]
Nov 02 17:38:15.054413 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.054392    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w6-7df526-openshift-default-dockercfg-cjrwk\""
Nov 02 17:38:15.123921 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.123890    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:15.142186 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.142158    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.142186 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.142189    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j79jc\" (UniqueName: \"kubernetes.io/projected/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-kube-api-access-j79jc\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.142411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.142212    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.142411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.142327    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.142411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.142355    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.142411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.142394    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.142553 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.142507    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.142594 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.142566    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.142629 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.142591    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.188364 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.188333    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"]
Nov 02 17:38:15.191573 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.191552    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.210951 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.210926    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-120394-openshift-default-dockercfg-qq496\""
Nov 02 17:38:15.215912 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.215887    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"]
Nov 02 17:38:15.243721 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.243688    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.243920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.243724    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.243920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.243747    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd44782-84e1-411a-9a1c-6d21456733bb-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.243920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.243882    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.244096 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.243936    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.244096 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.243969    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.244096 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244016    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd44782-84e1-411a-9a1c-6d21456733bb-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.244096 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244051    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.244267 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244176    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.244267 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244229    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.244267 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244256    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd44782-84e1-411a-9a1c-6d21456733bb-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.244419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244313    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.244419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244331    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.244419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244358    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j79jc\" (UniqueName: \"kubernetes.io/projected/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-kube-api-access-j79jc\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.244419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244399    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.244619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244412    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.244619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244440    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.244619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244472    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k7556\" (UniqueName: \"kubernetes.io/projected/acd44782-84e1-411a-9a1c-6d21456733bb-kube-api-access-k7556\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.244619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244520    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.244619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244554    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.244856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244622    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.244959 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.244938    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.245329 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.245299    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.246603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.246574    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.247254 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.247234    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.286325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.286251    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.286325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.286268    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j79jc\" (UniqueName: \"kubernetes.io/projected/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-kube-api-access-j79jc\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.336239 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.336207    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:15.346076 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.346047    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.346196 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.346101    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd44782-84e1-411a-9a1c-6d21456733bb-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.346196 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.346173    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd44782-84e1-411a-9a1c-6d21456733bb-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.346313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.346212    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.346313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.346243    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.346313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.346283    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.346313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.346309    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd44782-84e1-411a-9a1c-6d21456733bb-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.346523 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.346376    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.346523 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.346404    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k7556\" (UniqueName: \"kubernetes.io/projected/acd44782-84e1-411a-9a1c-6d21456733bb-kube-api-access-k7556\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.346634 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.346556    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.347163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.346927    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd44782-84e1-411a-9a1c-6d21456733bb-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.347163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.347006    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.347523 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.347208    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.347523 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.347230    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.348662 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.348604    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.350011 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.349992    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd44782-84e1-411a-9a1c-6d21456733bb-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.377318 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.377288    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd44782-84e1-411a-9a1c-6d21456733bb-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.378156 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.378121    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k7556\" (UniqueName: \"kubernetes.io/projected/acd44782-84e1-411a-9a1c-6d21456733bb-kube-api-access-k7556\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-j24dq\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.409754 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.409703    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:15.503951 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.503917    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:15.587257 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.587224    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:15.671463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.671437    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"]
Nov 02 17:38:15.673914 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:38:15.673877    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06c8d0c2_d05a_4ece_a583_bfa1aed218ce.slice/crio-7d47d1795bb8382ca32ce475cbee2f3d6c8dc511aa5b08da9e3750aa850b2be2 WatchSource:0}: Error finding container 7d47d1795bb8382ca32ce475cbee2f3d6c8dc511aa5b08da9e3750aa850b2be2: Status 404 returned error can't find the container with id 7d47d1795bb8382ca32ce475cbee2f3d6c8dc511aa5b08da9e3750aa850b2be2
Nov 02 17:38:15.675985 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.675953    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:15.676096 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.676028    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:15.676096 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.676074    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:15.744261 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.743967    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"]
Nov 02 17:38:15.749033 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.749010    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.753468 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.753442    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:15.765167 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.765147    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-069fa5-openshift-default-dockercfg-ckxtj\""
Nov 02 17:38:15.776368 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.775173    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"]
Nov 02 17:38:15.813638 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.813559    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"]
Nov 02 17:38:15.819721 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.819691    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:15.819832 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.819751    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:15.819832 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.819778    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:15.852299 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.852270    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.852445 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.852305    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.852445 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.852332    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.852445 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.852354    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.852617 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.852492    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.852617 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.852536    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.852617 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.852555    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.852778 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.852636    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cp2xr\" (UniqueName: \"kubernetes.io/projected/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-kube-api-access-cp2xr\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.852778 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.852714    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.889417 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.887436    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"]
Nov 02 17:38:15.891324 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.891297    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:15.911300 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.911254    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w2-f0a3b8-openshift-default-dockercfg-nch4k\""
Nov 02 17:38:15.929155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.925086    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"]
Nov 02 17:38:15.932651 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.932433    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:15.939740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.939704    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s" event={"ID":"06c8d0c2-d05a-4ece-a583-bfa1aed218ce","Type":"ContainerStarted","Data":"342f9a8ed8df1f4b2c22ebbefea71c3f8a51cdb1349cb8424e3f22803605ac5c"}
Nov 02 17:38:15.939891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.939761    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s" event={"ID":"06c8d0c2-d05a-4ece-a583-bfa1aed218ce","Type":"ContainerStarted","Data":"7d47d1795bb8382ca32ce475cbee2f3d6c8dc511aa5b08da9e3750aa850b2be2"}
Nov 02 17:38:15.941963 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.941939    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" event={"ID":"acd44782-84e1-411a-9a1c-6d21456733bb","Type":"ContainerStarted","Data":"c07d4b5f3cb6c757a1c1c8c0ec52b7bca550932f78aa3af4048216e01b4efc76"}
Nov 02 17:38:15.942070 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.941987    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" event={"ID":"acd44782-84e1-411a-9a1c-6d21456733bb","Type":"ContainerStarted","Data":"20ccacdbcdc8a5973d24d7c2aeedd34afed8581ce29954a8d6e0374ab30d9bf1"}
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.953772    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.953830    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.953876    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.953901    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.953917    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.953941    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.953957    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb933398-5485-4600-a38e-640531c54137-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.953975    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.953993    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.954012    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb933398-5485-4600-a38e-640531c54137-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.954047    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.954072    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb933398-5485-4600-a38e-640531c54137-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.954096    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.954110    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.954127    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.954142    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.954174    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cp2xr\" (UniqueName: \"kubernetes.io/projected/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-kube-api-access-cp2xr\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.954201    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t994b\" (UniqueName: \"kubernetes.io/projected/fb933398-5485-4600-a38e-640531c54137-kube-api-access-t994b\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.954902    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.955044    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.955643    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.956299    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.956834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.956580    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.962094 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.957158    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.962094 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.958475    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.981495 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.981464    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.981495 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.981473    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cp2xr\" (UniqueName: \"kubernetes.io/projected/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-kube-api-access-cp2xr\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:15.995099 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:15.995050    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" podStartSLOduration=0.995036717 podStartE2EDuration="995.036717ms" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:15.994612943 +0000 UTC m=+512.485676106" watchObservedRunningTime="2026-11-02 17:38:15.995036717 +0000 UTC m=+512.486099877"
Nov 02 17:38:16.052672 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.052567    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s" podStartSLOduration=2.052548046 podStartE2EDuration="2.052548046s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:16.051328025 +0000 UTC m=+512.542391215" watchObservedRunningTime="2026-11-02 17:38:16.052548046 +0000 UTC m=+512.543611220"
Nov 02 17:38:16.054976 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.054951    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.054976 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.054983    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.055220 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.055017    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t994b\" (UniqueName: \"kubernetes.io/projected/fb933398-5485-4600-a38e-640531c54137-kube-api-access-t994b\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.055220 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.055087    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.055220 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.055137    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.055220 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.055209    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb933398-5485-4600-a38e-640531c54137-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.055439 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.055242    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.055439 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.055282    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb933398-5485-4600-a38e-640531c54137-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.055439 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.055391    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb933398-5485-4600-a38e-640531c54137-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.056463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.056439    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.056955 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.056931    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.057374 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.057347    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.057621 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.057601    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb933398-5485-4600-a38e-640531c54137-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.057692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.057638    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.059210 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.059181    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.059296 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.059215    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb933398-5485-4600-a38e-640531c54137-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.064130 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.064074    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:16.084577 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.084541    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb933398-5485-4600-a38e-640531c54137-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.085625 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.085588    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t994b\" (UniqueName: \"kubernetes.io/projected/fb933398-5485-4600-a38e-640531c54137-kube-api-access-t994b\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.113561 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.113529    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:16.171992 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:38:16.171955    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:38:16.210278 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.210013    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:16.274585 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.274260    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:16.336674 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.336583    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:16.343228 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.343199    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:16.395211 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.395122    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"]
Nov 02 17:38:16.403461 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:38:16.403432    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae55ce0a_88e9_4ad4_bfdd_f55dbd9389b0.slice/crio-2731fa237307e8ca02049b0c93ee8edf1340ab68b0f85380c97410539d6ab10d WatchSource:0}: Error finding container 2731fa237307e8ca02049b0c93ee8edf1340ab68b0f85380c97410539d6ab10d: Status 404 returned error can't find the container with id 2731fa237307e8ca02049b0c93ee8edf1340ab68b0f85380c97410539d6ab10d
Nov 02 17:38:16.406132 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.406087    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:16.406222 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.406177    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:16.406264 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.406220    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:16.425205 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.425170    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:16.504576 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.504545    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:16.510309 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.510284    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:16.557047 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.557018    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"]
Nov 02 17:38:16.559978 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:38:16.559943    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb933398_5485_4600_a38e_640531c54137.slice/crio-ec06b06c2158e74a662d63cf5e79c65b848abe3e82ccc2b7339d79f5257d9e96 WatchSource:0}: Error finding container ec06b06c2158e74a662d63cf5e79c65b848abe3e82ccc2b7339d79f5257d9e96: Status 404 returned error can't find the container with id ec06b06c2158e74a662d63cf5e79c65b848abe3e82ccc2b7339d79f5257d9e96
Nov 02 17:38:16.562784 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.562749    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:16.562919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.562901    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:16.562975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.562942    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:16.950552 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.950519    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" event={"ID":"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0","Type":"ContainerStarted","Data":"a1b54e6df78108d7c52226ef6abfb9dcf23c4f7a7ff0b1f2e044b8ac4afdddec"}
Nov 02 17:38:16.951029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.950576    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" event={"ID":"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0","Type":"ContainerStarted","Data":"2731fa237307e8ca02049b0c93ee8edf1340ab68b0f85380c97410539d6ab10d"}
Nov 02 17:38:16.952709 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.952675    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr" event={"ID":"fb933398-5485-4600-a38e-640531c54137","Type":"ContainerStarted","Data":"efc584ce44ecf1a685d22ebf1c76e1b3706e98d16b83773fa74439aab3a7eb04"}
Nov 02 17:38:16.952899 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.952733    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr" event={"ID":"fb933398-5485-4600-a38e-640531c54137","Type":"ContainerStarted","Data":"ec06b06c2158e74a662d63cf5e79c65b848abe3e82ccc2b7339d79f5257d9e96"}
Nov 02 17:38:16.953210 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.953109    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:16.953210 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.953167    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:16.953915 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.953897    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:16.954007 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:16.953971    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:17.006481 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:17.006423    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" podStartSLOduration=2.006409616 podStartE2EDuration="2.006409616s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:17.004909329 +0000 UTC m=+513.495972527" watchObservedRunningTime="2026-11-02 17:38:17.006409616 +0000 UTC m=+513.497472837"
Nov 02 17:38:17.064636 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:17.064598    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:17.069980 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:17.069945    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:17.126039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:17.125989    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr" podStartSLOduration=2.125973283 podStartE2EDuration="2.125973283s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:17.124326218 +0000 UTC m=+513.615389382" watchObservedRunningTime="2026-11-02 17:38:17.125973283 +0000 UTC m=+513.617036444"
Nov 02 17:38:17.215895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:17.215799    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:17.220738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:17.220712    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:17.957582 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:17.957548    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:17.958066 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:17.957924    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:17.958528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:17.958507    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:17.958625 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:17.958557    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:18.740110 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:18.740071    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:18.909256 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:18.909223    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:19.066990 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:19.066904    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:19.290696 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:19.290666    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:19.529129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:19.529096    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:19.745348 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:19.745312    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:19.945726 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:19.945695    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:20.144302 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:20.144245    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:20.473308 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:20.473258    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:20.667489 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:20.667456    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:21.206970 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:21.206918    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:21.383353 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:21.383318    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:21.596147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:21.596059    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:21.756449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:21.756415    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:22.777145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:22.777107    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:22.943728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:22.943698    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:23.093437 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:23.093343    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:23.735021 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:23.734967    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:23.893195 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:23.893156    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:25.128786 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:25.128735    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:26.221489 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:26.221447    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:26.372171 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:26.372134    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:27.107253 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.107218    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:27.231838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.227304    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"]
Nov 02 17:38:27.246836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.246101    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"
Nov 02 17:38:27.273275 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.272692    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"]
Nov 02 17:38:27.275410 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.275383    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-f0a3b8-dockercfg-h2cjr\""
Nov 02 17:38:27.275636 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.275612    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-f0a3b8\""
Nov 02 17:38:27.276942 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.276919    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:27.332253 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.332221    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"]
Nov 02 17:38:27.337446 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.337423    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"
Nov 02 17:38:27.353467 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.353421    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"]
Nov 02 17:38:27.383064 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.383028    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2d9tp\" (UniqueName: \"kubernetes.io/projected/b5d45c38-1cef-4aa9-a519-98aed00e8727-kube-api-access-2d9tp\") pod \"payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz\" (UID: \"b5d45c38-1cef-4aa9-a519-98aed00e8727\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"
Nov 02 17:38:27.383245 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.383206    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d45c38-1cef-4aa9-a519-98aed00e8727-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz\" (UID: \"b5d45c38-1cef-4aa9-a519-98aed00e8727\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"
Nov 02 17:38:27.484112 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.484071    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d45c38-1cef-4aa9-a519-98aed00e8727-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz\" (UID: \"b5d45c38-1cef-4aa9-a519-98aed00e8727\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"
Nov 02 17:38:27.484301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.484130    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dvhdh\" (UniqueName: \"kubernetes.io/projected/18a32a7b-66fa-4f34-ab36-2beb0169e430-kube-api-access-dvhdh\") pod \"payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4\" (UID: \"18a32a7b-66fa-4f34-ab36-2beb0169e430\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"
Nov 02 17:38:27.484301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.484249    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2d9tp\" (UniqueName: \"kubernetes.io/projected/b5d45c38-1cef-4aa9-a519-98aed00e8727-kube-api-access-2d9tp\") pod \"payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz\" (UID: \"b5d45c38-1cef-4aa9-a519-98aed00e8727\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"
Nov 02 17:38:27.484417 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.484390    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a32a7b-66fa-4f34-ab36-2beb0169e430-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4\" (UID: \"18a32a7b-66fa-4f34-ab36-2beb0169e430\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"
Nov 02 17:38:27.484894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.484877    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d45c38-1cef-4aa9-a519-98aed00e8727-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz\" (UID: \"b5d45c38-1cef-4aa9-a519-98aed00e8727\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"
Nov 02 17:38:27.519440 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.519411    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2d9tp\" (UniqueName: \"kubernetes.io/projected/b5d45c38-1cef-4aa9-a519-98aed00e8727-kube-api-access-2d9tp\") pod \"payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz\" (UID: \"b5d45c38-1cef-4aa9-a519-98aed00e8727\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"
Nov 02 17:38:27.585166 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.585126    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a32a7b-66fa-4f34-ab36-2beb0169e430-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4\" (UID: \"18a32a7b-66fa-4f34-ab36-2beb0169e430\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"
Nov 02 17:38:27.585339 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.585202    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dvhdh\" (UniqueName: \"kubernetes.io/projected/18a32a7b-66fa-4f34-ab36-2beb0169e430-kube-api-access-dvhdh\") pod \"payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4\" (UID: \"18a32a7b-66fa-4f34-ab36-2beb0169e430\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"
Nov 02 17:38:27.585908 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.585881    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a32a7b-66fa-4f34-ab36-2beb0169e430-plugins-config-volume\") pod \"payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4\" (UID: \"18a32a7b-66fa-4f34-ab36-2beb0169e430\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"
Nov 02 17:38:27.589952 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.589909    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"
Nov 02 17:38:27.611238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.611206    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dvhdh\" (UniqueName: \"kubernetes.io/projected/18a32a7b-66fa-4f34-ab36-2beb0169e430-kube-api-access-dvhdh\") pod \"payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4\" (UID: \"18a32a7b-66fa-4f34-ab36-2beb0169e430\") " pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"
Nov 02 17:38:27.686534 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.686497    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"
Nov 02 17:38:27.958904 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:27.958542    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"]
Nov 02 17:38:28.018571 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:28.018532    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" event={"ID":"b5d45c38-1cef-4aa9-a519-98aed00e8727","Type":"ContainerStarted","Data":"0183eddcebc7b4d46fa8e5bbf2be7bd6ff098f83a45e201f0cb7a81c25790914"}
Nov 02 17:38:28.025067 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:28.025037    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"]
Nov 02 17:38:28.029429 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:38:28.029400    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18a32a7b_66fa_4f34_ab36_2beb0169e430.slice/crio-695b8b70dea01fa843ea28ef2b7e9e8b64d8246dc2edfad876806e3044f16058 WatchSource:0}: Error finding container 695b8b70dea01fa843ea28ef2b7e9e8b64d8246dc2edfad876806e3044f16058: Status 404 returned error can't find the container with id 695b8b70dea01fa843ea28ef2b7e9e8b64d8246dc2edfad876806e3044f16058
Nov 02 17:38:28.825277 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:28.824570    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:29.019290 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:29.019254    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:29.029623 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:29.029594    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" event={"ID":"18a32a7b-66fa-4f34-ab36-2beb0169e430","Type":"ContainerStarted","Data":"695b8b70dea01fa843ea28ef2b7e9e8b64d8246dc2edfad876806e3044f16058"}
Nov 02 17:38:29.210013 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:29.209338    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:29.401853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:29.399271    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:30.548483 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.548449    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:30.710910 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.710864    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"]
Nov 02 17:38:30.724014 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.723983    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"
Nov 02 17:38:30.727117 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.727088    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"]
Nov 02 17:38:30.734961 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.733666    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ae7b61-cd5f-4a38-b038-28aaa6991eb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j\" (UID: \"e5ae7b61-cd5f-4a38-b038-28aaa6991eb7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"
Nov 02 17:38:30.734961 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.733756    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzv9g\" (UniqueName: \"kubernetes.io/projected/e5ae7b61-cd5f-4a38-b038-28aaa6991eb7-kube-api-access-wzv9g\") pod \"payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j\" (UID: \"e5ae7b61-cd5f-4a38-b038-28aaa6991eb7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"
Nov 02 17:38:30.741414 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.741336    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:30.751036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.751016    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-2899ab-dockercfg-lwvws\""
Nov 02 17:38:30.751250 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.751212    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-2899ab\""
Nov 02 17:38:30.813432 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.813347    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"]
Nov 02 17:38:30.818272 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.818247    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 17:38:30.823751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.823726    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"]
Nov 02 17:38:30.834977 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.834407    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8tqd5\" (UniqueName: \"kubernetes.io/projected/bca2dabc-94e2-4875-ba3f-fa9332afe8dc-kube-api-access-8tqd5\") pod \"payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj\" (UID: \"bca2dabc-94e2-4875-ba3f-fa9332afe8dc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 17:38:30.834977 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.834469    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ae7b61-cd5f-4a38-b038-28aaa6991eb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j\" (UID: \"e5ae7b61-cd5f-4a38-b038-28aaa6991eb7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"
Nov 02 17:38:30.834977 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.834567    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca2dabc-94e2-4875-ba3f-fa9332afe8dc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj\" (UID: \"bca2dabc-94e2-4875-ba3f-fa9332afe8dc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 17:38:30.834977 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.834619    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzv9g\" (UniqueName: \"kubernetes.io/projected/e5ae7b61-cd5f-4a38-b038-28aaa6991eb7-kube-api-access-wzv9g\") pod \"payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j\" (UID: \"e5ae7b61-cd5f-4a38-b038-28aaa6991eb7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"
Nov 02 17:38:30.835337 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.835296    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ae7b61-cd5f-4a38-b038-28aaa6991eb7-plugins-config-volume\") pod \"payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j\" (UID: \"e5ae7b61-cd5f-4a38-b038-28aaa6991eb7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"
Nov 02 17:38:30.863294 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.861169    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzv9g\" (UniqueName: \"kubernetes.io/projected/e5ae7b61-cd5f-4a38-b038-28aaa6991eb7-kube-api-access-wzv9g\") pod \"payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j\" (UID: \"e5ae7b61-cd5f-4a38-b038-28aaa6991eb7\") " pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"
Nov 02 17:38:30.935777 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.935744    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca2dabc-94e2-4875-ba3f-fa9332afe8dc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj\" (UID: \"bca2dabc-94e2-4875-ba3f-fa9332afe8dc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 17:38:30.935981 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.935871    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8tqd5\" (UniqueName: \"kubernetes.io/projected/bca2dabc-94e2-4875-ba3f-fa9332afe8dc-kube-api-access-8tqd5\") pod \"payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj\" (UID: \"bca2dabc-94e2-4875-ba3f-fa9332afe8dc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 17:38:30.936886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.936861    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca2dabc-94e2-4875-ba3f-fa9332afe8dc-plugins-config-volume\") pod \"payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj\" (UID: \"bca2dabc-94e2-4875-ba3f-fa9332afe8dc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 17:38:30.975063 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.975025    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:31.003839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:30.998506    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8tqd5\" (UniqueName: \"kubernetes.io/projected/bca2dabc-94e2-4875-ba3f-fa9332afe8dc-kube-api-access-8tqd5\") pod \"payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj\" (UID: \"bca2dabc-94e2-4875-ba3f-fa9332afe8dc\") " pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 17:38:31.044280 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:31.044249    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"
Nov 02 17:38:31.135842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:31.130466    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:31.138573 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:31.138288    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 17:38:31.412955 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:31.412861    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"]
Nov 02 17:38:31.418624 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:38:31.418591    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5ae7b61_cd5f_4a38_b038_28aaa6991eb7.slice/crio-5a4683ca1335bb45dee0f2972033c98eb26ce6dc1e172bf2a343ed5d8e4bc16c WatchSource:0}: Error finding container 5a4683ca1335bb45dee0f2972033c98eb26ce6dc1e172bf2a343ed5d8e4bc16c: Status 404 returned error can't find the container with id 5a4683ca1335bb45dee0f2972033c98eb26ce6dc1e172bf2a343ed5d8e4bc16c
Nov 02 17:38:31.604076 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:31.574655    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"]
Nov 02 17:38:31.604076 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:38:31.577772    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbca2dabc_94e2_4875_ba3f_fa9332afe8dc.slice/crio-8ef77d511cfd1d75eff7c29a150cd6acf17c03b9711c64acffc4f36a9e09d350 WatchSource:0}: Error finding container 8ef77d511cfd1d75eff7c29a150cd6acf17c03b9711c64acffc4f36a9e09d350: Status 404 returned error can't find the container with id 8ef77d511cfd1d75eff7c29a150cd6acf17c03b9711c64acffc4f36a9e09d350
Nov 02 17:38:32.055589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:32.055558    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" event={"ID":"b5d45c38-1cef-4aa9-a519-98aed00e8727","Type":"ContainerStarted","Data":"fb0803029471e70042acd1e45626801c9ff68565bd8c57f88a3acdbb82e2b010"}
Nov 02 17:38:32.055929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:32.055876    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"
Nov 02 17:38:32.058530 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:32.058507    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" event={"ID":"bca2dabc-94e2-4875-ba3f-fa9332afe8dc","Type":"ContainerStarted","Data":"8ef77d511cfd1d75eff7c29a150cd6acf17c03b9711c64acffc4f36a9e09d350"}
Nov 02 17:38:32.059032 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:32.059012    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 17:38:32.060033 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:32.060011    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j" event={"ID":"e5ae7b61-cd5f-4a38-b038-28aaa6991eb7","Type":"ContainerStarted","Data":"09f75adb956ba8d1b347af752e4d6d17af18d67cc64d9c62d56a97f64f5dde8e"}
Nov 02 17:38:32.060117 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:32.060052    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j" event={"ID":"e5ae7b61-cd5f-4a38-b038-28aaa6991eb7","Type":"ContainerStarted","Data":"5a4683ca1335bb45dee0f2972033c98eb26ce6dc1e172bf2a343ed5d8e4bc16c"}
Nov 02 17:38:32.060438 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:32.060418    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"
Nov 02 17:38:32.110888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:32.107446    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podStartSLOduration=1.2264569810000001 podStartE2EDuration="5.107427337s" totalImagesPullingTime="3.880970356s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:32.104361675 +0000 UTC m=+528.595424861" watchObservedRunningTime="2026-11-02 17:38:32.107427337 +0000 UTC m=+528.598490498"
Nov 02 17:38:32.158588 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:32.158302    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" podStartSLOduration=1.812257894 podStartE2EDuration="2.158276994s" totalImagesPullingTime="346.0191ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:32.156120591 +0000 UTC m=+528.647183775" watchObservedRunningTime="2026-11-02 17:38:32.158276994 +0000 UTC m=+528.649340155"
Nov 02 17:38:32.960748 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:32.960718    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:33.080867 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:33.078442    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" event={"ID":"18a32a7b-66fa-4f34-ab36-2beb0169e430","Type":"ContainerStarted","Data":"65dc1c7055ca0c593790169b343803715577f608817bf5ba500657f16af44d0b"}
Nov 02 17:38:33.080867 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:33.079391    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"
Nov 02 17:38:33.090839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:33.086872    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" event={"ID":"bca2dabc-94e2-4875-ba3f-fa9332afe8dc","Type":"ContainerStarted","Data":"7c1de68e597eb58fefb3cd933a0ac134632987153b7d92161a06b331b295e194"}
Nov 02 17:38:33.128418 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:33.128307    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:33.129394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:33.129054    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j" podStartSLOduration=2.708276233 podStartE2EDuration="3.12904239s" totalImagesPullingTime="420.766157ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:30 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:32.21526381 +0000 UTC m=+528.706326968" watchObservedRunningTime="2026-11-02 17:38:33.12904239 +0000 UTC m=+529.620105578"
Nov 02 17:38:33.130042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:33.130005    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podStartSLOduration=2.207604192 podStartE2EDuration="6.129995654s" totalImagesPullingTime="3.922391462s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:33.127924531 +0000 UTC m=+529.618987695" watchObservedRunningTime="2026-11-02 17:38:33.129995654 +0000 UTC m=+529.621058815"
Nov 02 17:38:35.121833 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:35.121773    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:36.819212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:36.816725    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"]
Nov 02 17:38:36.826263 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:36.826236    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"]
Nov 02 17:38:36.826409 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:36.826388    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:36.842731 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:36.842042    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 17:38:36.842893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:36.842852    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 17:38:36.854129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:36.854105    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 17:38:36.854129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:36.854115    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-f0a3b8-serving-cert\""
Nov 02 17:38:36.854322 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:36.854210    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gml5d\""
Nov 02 17:38:36.912860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:36.912806    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:36.929653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:36.929624    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ebd3467-841a-4331-b149-8acf6e60c12c-metrics-certs\") pod \"maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:36.929803 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:36.929676    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebd3467-841a-4331-b149-8acf6e60c12c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:36.929908 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:36.929885    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ebd3467-841a-4331-b149-8acf6e60c12c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:36.929944 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:36.929925    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64jnz\" (UniqueName: \"kubernetes.io/projected/6ebd3467-841a-4331-b149-8acf6e60c12c-kube-api-access-64jnz\") pod \"maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:37.031769 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:37.031679    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ebd3467-841a-4331-b149-8acf6e60c12c-metrics-certs\") pod \"maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:37.031968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:37.031770    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebd3467-841a-4331-b149-8acf6e60c12c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:37.031968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:37.031949    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ebd3467-841a-4331-b149-8acf6e60c12c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:37.032085 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:37.031985    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64jnz\" (UniqueName: \"kubernetes.io/projected/6ebd3467-841a-4331-b149-8acf6e60c12c-kube-api-access-64jnz\") pod \"maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:37.032686 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:37.032633    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ebd3467-841a-4331-b149-8acf6e60c12c-openshift-service-ca\") pod \"maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:37.034359 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:37.034335    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ebd3467-841a-4331-b149-8acf6e60c12c-metrics-certs\") pod \"maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:37.034574 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:37.034554    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebd3467-841a-4331-b149-8acf6e60c12c-maas-api-tls\") pod \"maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:37.056024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:37.055991    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64jnz\" (UniqueName: \"kubernetes.io/projected/6ebd3467-841a-4331-b149-8acf6e60c12c-kube-api-access-64jnz\") pod \"maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:37.062737 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:37.062712    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:37.148790 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:37.148747    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:37.490750 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:37.490655    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"]
Nov 02 17:38:37.514328 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:38:37.514289    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebd3467_841a_4331_b149_8acf6e60c12c.slice/crio-3effdad3d073c6daba1da297c159b41a5a9f572263586cea3824001afa765a92 WatchSource:0}: Error finding container 3effdad3d073c6daba1da297c159b41a5a9f572263586cea3824001afa765a92: Status 404 returned error can't find the container with id 3effdad3d073c6daba1da297c159b41a5a9f572263586cea3824001afa765a92
Nov 02 17:38:38.132309 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:38.132223    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" event={"ID":"6ebd3467-841a-4331-b149-8acf6e60c12c","Type":"ContainerStarted","Data":"3effdad3d073c6daba1da297c159b41a5a9f572263586cea3824001afa765a92"}
Nov 02 17:38:38.134614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:38.134581    4456 generic.go:356] "Generic (PLEG): container finished" podID="acd44782-84e1-411a-9a1c-6d21456733bb" containerID="c07d4b5f3cb6c757a1c1c8c0ec52b7bca550932f78aa3af4048216e01b4efc76" exitCode=0
Nov 02 17:38:38.134845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:38.134768    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" event={"ID":"acd44782-84e1-411a-9a1c-6d21456733bb","Type":"ContainerDied","Data":"c07d4b5f3cb6c757a1c1c8c0ec52b7bca550932f78aa3af4048216e01b4efc76"}
Nov 02 17:38:38.135728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:38.135711    4456 scope.go:122] "RemoveContainer" containerID="c07d4b5f3cb6c757a1c1c8c0ec52b7bca550932f78aa3af4048216e01b4efc76"
Nov 02 17:38:38.138023 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:38.137431    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:38.138023 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:38.137499    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:38.138023 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:38.137529    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:38.840525 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:38.840494    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:39.046915 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:39.046792    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:39.151952 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:39.151846    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" event={"ID":"acd44782-84e1-411a-9a1c-6d21456733bb","Type":"ContainerStarted","Data":"5d1811fa76419aa50beae48550ad5ab8de1c9137d28d84239f25338279ddf29d"}
Nov 02 17:38:39.443076 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:39.443026    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:39.505841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:39.505176    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:39.505841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:39.505313    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 17:38:39.505841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:39.505360    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 17:38:39.640866 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:39.640399    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:40.510842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:40.507303    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 17:38:40.510842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:40.507387    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 17:38:40.998857 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:40.989386    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:41.166752 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:41.166698    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" event={"ID":"6ebd3467-841a-4331-b149-8acf6e60c12c","Type":"ContainerStarted","Data":"e4e31f901eface112badee8c3415cf9f3881ca46d033fd3f068c3ec2f49f9ab9"}
Nov 02 17:38:41.166973 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:41.166783    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:41.220429 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:41.220369    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podStartSLOduration=2.110663548 podStartE2EDuration="5.220353897s" totalImagesPullingTime="3.109690349s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:36 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:41.219170094 +0000 UTC m=+537.710233292" watchObservedRunningTime="2026-11-02 17:38:41.220353897 +0000 UTC m=+537.711417104"
Nov 02 17:38:41.507755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:41.507276    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 17:38:41.507755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:41.507344    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 17:38:42.150122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.148489    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"]
Nov 02 17:38:42.152563 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.152531    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.188050 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.188015    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"simulator-w1-120394-kserve-self-signed-certs\""
Nov 02 17:38:42.188708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.188381    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"kube-root-ca.crt\""
Nov 02 17:38:42.188915 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.188523    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"default-dockercfg-8x5zb\""
Nov 02 17:38:42.189036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.188637    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"openshift-service-ca.crt\""
Nov 02 17:38:42.213440 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.213408    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"]
Nov 02 17:38:42.241930 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.241888    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-home\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.242147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.242069    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-tmp-dir\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.242147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.242121    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-model-cache\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.242280 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.242150    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4dtmz\" (UniqueName: \"kubernetes.io/projected/5734524d-67b2-4e24-a836-3ac078d5a271-kube-api-access-4dtmz\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.242280 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.242255    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5734524d-67b2-4e24-a836-3ac078d5a271-tls-certs\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.242451 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.242424    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-dshm\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.243701 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.243668    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:42.343958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.343912    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-tmp-dir\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.344155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.343977    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-model-cache\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.344155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.344005    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4dtmz\" (UniqueName: \"kubernetes.io/projected/5734524d-67b2-4e24-a836-3ac078d5a271-kube-api-access-4dtmz\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.344155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.344044    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5734524d-67b2-4e24-a836-3ac078d5a271-tls-certs\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.344155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.344112    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-dshm\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.344155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.344154    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-home\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.344433 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.344252    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-tmp-dir\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.344490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.344478    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-home\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.344744 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.344705    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-model-cache\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.346937 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.346910    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-dshm\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.347531 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.347511    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5734524d-67b2-4e24-a836-3ac078d5a271-tls-certs\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.368928 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.368904    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4dtmz\" (UniqueName: \"kubernetes.io/projected/5734524d-67b2-4e24-a836-3ac078d5a271-kube-api-access-4dtmz\") pod \"simulator-w1-120394-kserve-6979bf4f4f-scg5b\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") " pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.468170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.467933    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:42.510322 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.510279    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 17:38:42.510570 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.510543    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 17:38:42.820585 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.820540    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"]
Nov 02 17:38:42.823449 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:38:42.823411    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5734524d_67b2_4e24_a836_3ac078d5a271.slice/crio-73d2c013b474c6ff1af7f2714382fc5912e92bcd9b6bb918cb97fe9b6363856b WatchSource:0}: Error finding container 73d2c013b474c6ff1af7f2714382fc5912e92bcd9b6bb918cb97fe9b6363856b: Status 404 returned error can't find the container with id 73d2c013b474c6ff1af7f2714382fc5912e92bcd9b6bb918cb97fe9b6363856b
Nov 02 17:38:42.860009 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:42.859955    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:43.110889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:43.108013    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"
Nov 02 17:38:43.213111 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:43.213061    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" event={"ID":"5734524d-67b2-4e24-a836-3ac078d5a271","Type":"ContainerStarted","Data":"764623736da34bdff5ec0d93b5511172b9e7e7729309eb85acb60f34966d682b"}
Nov 02 17:38:43.213615 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:43.213125    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" event={"ID":"5734524d-67b2-4e24-a836-3ac078d5a271","Type":"ContainerStarted","Data":"73d2c013b474c6ff1af7f2714382fc5912e92bcd9b6bb918cb97fe9b6363856b"}
Nov 02 17:38:43.257313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:43.257256    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podStartSLOduration=1.2572420100000001 podStartE2EDuration="1.25724201s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:38:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:38:43.256998825 +0000 UTC m=+539.748061987" watchObservedRunningTime="2026-11-02 17:38:43.25724201 +0000 UTC m=+539.748305176"
Nov 02 17:38:43.468595 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:43.468548    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:43.511846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:43.506235    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 17:38:43.511846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:43.506327    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 17:38:44.141866 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:44.129056    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 17:38:44.141866 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:44.129373    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"
Nov 02 17:38:44.506839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:44.506604    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused" start-of-body=
Nov 02 17:38:44.506839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:44.506670    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": dial tcp 10.134.0.39:15021: connect: connection refused"
Nov 02 17:38:44.789408 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:44.785185    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:45.138251 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:45.138158    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"
Nov 02 17:38:45.540685 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:45.539148    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:45.540685 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:45.540534    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:45.547615 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:45.547538    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 17:38:46.167322 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:38:46.167289    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:38:46.355966 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:46.355050    4456 generic.go:356] "Generic (PLEG): container finished" podID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" containerID="342f9a8ed8df1f4b2c22ebbefea71c3f8a51cdb1349cb8424e3f22803605ac5c" exitCode=0
Nov 02 17:38:46.355966 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:46.355498    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s" event={"ID":"06c8d0c2-d05a-4ece-a583-bfa1aed218ce","Type":"ContainerDied","Data":"342f9a8ed8df1f4b2c22ebbefea71c3f8a51cdb1349cb8424e3f22803605ac5c"}
Nov 02 17:38:46.355966 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:46.355954    4456 scope.go:122] "RemoveContainer" containerID="342f9a8ed8df1f4b2c22ebbefea71c3f8a51cdb1349cb8424e3f22803605ac5c"
Nov 02 17:38:46.357450 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:46.357420    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:46.357567 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:46.357512    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:46.357631 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:46.357571    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:46.921684 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:46.921596    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:46.953355 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:46.953316    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:47.193841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:47.193041    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 17:38:47.377525 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:47.377474    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s" event={"ID":"06c8d0c2-d05a-4ece-a583-bfa1aed218ce","Type":"ContainerStarted","Data":"b6c29b54d2e8809522ee97f272b9dfd8c889bc72413ca91c1099b08a1dd31041"}
Nov 02 17:38:48.338565 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:48.338526    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:48.341324 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:48.341297    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 17:38:48.341500 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:48.341476    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s" podUID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 17:38:48.962838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:48.962764    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:49.158844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:49.158269    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:49.344895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:49.341222    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 17:38:49.344895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:49.341283    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s" podUID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 17:38:49.742599 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:49.742564    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:50.015894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:50.007666    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:50.177666 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:50.177591    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:50.337519 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:50.337431    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 17:38:50.337519 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:50.337500    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s" podUID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 17:38:50.369345 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:50.369311    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:50.528898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:50.528851    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:50.685459 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:50.685391    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:51.087542 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:51.087341    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:51.303411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:51.303332    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:51.338894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:51.337600    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 17:38:51.338894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:51.337661    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s" podUID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 17:38:51.462454 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:51.462366    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:51.639134 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:51.638964    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:52.337942 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:52.337439    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused" start-of-body=
Nov 02 17:38:52.337942 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:52.337509    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s" podUID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.38:15021/healthz/ready\": dial tcp 10.134.0.38:15021: connect: connection refused"
Nov 02 17:38:52.469516 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:52.469484    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:52.547594 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:52.547567    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:53.271269 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:53.267262    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:53.351513 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:53.350091    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:53.351513 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:53.351455    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:53.370317 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:53.370241    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:38:53.467534 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:53.467500    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:38:53.537699 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:53.537449    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:54.456327 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:54.456300    4456 generic.go:356] "Generic (PLEG): container finished" podID="fb933398-5485-4600-a38e-640531c54137" containerID="efc584ce44ecf1a685d22ebf1c76e1b3706e98d16b83773fa74439aab3a7eb04" exitCode=0
Nov 02 17:38:54.456482 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:54.456368    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr" event={"ID":"fb933398-5485-4600-a38e-640531c54137","Type":"ContainerDied","Data":"efc584ce44ecf1a685d22ebf1c76e1b3706e98d16b83773fa74439aab3a7eb04"}
Nov 02 17:38:54.456985 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:54.456969    4456 scope.go:122] "RemoveContainer" containerID="efc584ce44ecf1a685d22ebf1c76e1b3706e98d16b83773fa74439aab3a7eb04"
Nov 02 17:38:54.458206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:54.458180    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:54.458287 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:54.458245    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:54.458287 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:54.458275    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:54.958024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:54.957993    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:55.171337 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:55.171301    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:55.463853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:55.463804    4456 generic.go:356] "Generic (PLEG): container finished" podID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerID="a1b54e6df78108d7c52226ef6abfb9dcf23c4f7a7ff0b1f2e044b8ac4afdddec" exitCode=0
Nov 02 17:38:55.463853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:55.463846    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" event={"ID":"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0","Type":"ContainerDied","Data":"a1b54e6df78108d7c52226ef6abfb9dcf23c4f7a7ff0b1f2e044b8ac4afdddec"}
Nov 02 17:38:55.464320 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:55.464285    4456 scope.go:122] "RemoveContainer" containerID="a1b54e6df78108d7c52226ef6abfb9dcf23c4f7a7ff0b1f2e044b8ac4afdddec"
Nov 02 17:38:55.465844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:55.465789    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:55.465957 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:55.465882    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:55.465957 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:55.465919    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:38:55.466247 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:55.466227    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr" event={"ID":"fb933398-5485-4600-a38e-640531c54137","Type":"ContainerStarted","Data":"c87a2115f023acd478f1d6c28935f520f81e705ba4d79b759e5802589ddebe1c"}
Nov 02 17:38:55.647705 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:55.647658    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:55.818897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:55.818594    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:55.972716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:55.972686    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:56.215960 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:56.215924    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:38:56.216154 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:56.216070    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 17:38:56.216154 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:56.216119    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr" podUID="fb933398-5485-4600-a38e-640531c54137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 17:38:56.479059 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:56.478970    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" event={"ID":"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0","Type":"ContainerStarted","Data":"97a86bee14d24f04dff0a2c6c82d05fab51870580f831e892bca987e63a5078c"}
Nov 02 17:38:57.064671 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:57.064637    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:38:57.065138 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:57.064839    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 17:38:57.065138 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:57.064888    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 17:38:57.218841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:57.218394    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 17:38:57.218841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:57.218475    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr" podUID="fb933398-5485-4600-a38e-640531c54137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 17:38:57.456849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:57.452731    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:38:58.064899 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:58.064865    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 17:38:58.065314 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:58.064938    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 17:38:58.225841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:58.223197    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 17:38:58.225841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:58.223263    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr" podUID="fb933398-5485-4600-a38e-640531c54137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 17:38:59.065315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:59.065286    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 17:38:59.065687 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:59.065336    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 17:38:59.217840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:59.217291    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 17:38:59.217840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:59.217353    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr" podUID="fb933398-5485-4600-a38e-640531c54137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 17:38:59.375463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:38:59.375371    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:00.064275 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:00.064246    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 17:39:00.064443 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:00.064302    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 17:39:00.220841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:00.220218    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused" start-of-body=
Nov 02 17:39:00.220841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:00.220277    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr" podUID="fb933398-5485-4600-a38e-640531c54137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": dial tcp 10.134.0.41:15021: connect: connection refused"
Nov 02 17:39:00.349922 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:00.349808    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:00.508563 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:00.508514    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:01.017436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:01.017400    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:01.064909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:01.064878    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 17:39:01.065060 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:01.064932    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 17:39:01.210855 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:01.207396    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:01.235190 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:01.235157    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:39:01.236600 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:01.236576    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:39:01.237335 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:01.237312    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:39:01.508840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:01.508352    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:02.065213 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.065176    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused" start-of-body=
Nov 02 17:39:02.065405 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.065231    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": dial tcp 10.134.0.40:15021: connect: connection refused"
Nov 02 17:39:02.740104 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.740068    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"]
Nov 02 17:39:02.742872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.742850    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.777270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.777245    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"kube-root-ca.crt\""
Nov 02 17:39:02.777414 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.777396    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"simulator-w3-5eedab-kserve-self-signed-certs\""
Nov 02 17:39:02.777458 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.777402    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"default-dockercfg-q6667\""
Nov 02 17:39:02.777458 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.777445    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"openshift-service-ca.crt\""
Nov 02 17:39:02.789427 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.789401    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"]
Nov 02 17:39:02.857273 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.857208    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:02.861481 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.861455    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-dshm\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.861622 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.861487    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sns45\" (UniqueName: \"kubernetes.io/projected/0c6cec58-b8d9-4856-9446-811407a747ab-kube-api-access-sns45\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.861622 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.861508    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-tmp-dir\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.861715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.861697    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6cec58-b8d9-4856-9446-811407a747ab-tls-certs\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.861755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.861732    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-home\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.861809 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.861752    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-model-cache\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.957777 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.957744    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:39:02.963270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.963252    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6cec58-b8d9-4856-9446-811407a747ab-tls-certs\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.963344 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.963289    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-home\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.963344 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.963308    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-model-cache\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.963451 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.963344    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-dshm\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.963707 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.963683    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-home\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.963800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.963768    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-model-cache\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.963893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.963800    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sns45\" (UniqueName: \"kubernetes.io/projected/0c6cec58-b8d9-4856-9446-811407a747ab-kube-api-access-sns45\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.963893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.963866    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-tmp-dir\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.964345 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.964324    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-tmp-dir\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.965790 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.965765    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-dshm\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:02.965966 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:02.965949    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6cec58-b8d9-4856-9446-811407a747ab-tls-certs\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:03.006646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:03.006560    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sns45\" (UniqueName: \"kubernetes.io/projected/0c6cec58-b8d9-4856-9446-811407a747ab-kube-api-access-sns45\") pod \"simulator-w3-5eedab-kserve-795c4575bb-f25x5\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") " pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:03.053518 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:03.053481    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:03.069865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:03.069839    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:39:03.359467 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:03.359438    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"]
Nov 02 17:39:03.361204 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:39:03.361173    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0c6cec58_b8d9_4856_9446_811407a747ab.slice/crio-01a5b69e8c051a84622bfd5f598c65f6ed739617d900941ec81338e51762888e WatchSource:0}: Error finding container 01a5b69e8c051a84622bfd5f598c65f6ed739617d900941ec81338e51762888e: Status 404 returned error can't find the container with id 01a5b69e8c051a84622bfd5f598c65f6ed739617d900941ec81338e51762888e
Nov 02 17:39:03.494515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:03.494478    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:03.556341 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:03.556302    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" event={"ID":"0c6cec58-b8d9-4856-9446-811407a747ab","Type":"ContainerStarted","Data":"dee185d86f1b035d22e9bbaa40ba82863efde6c7b75aef6a674103522b0c63a9"}
Nov 02 17:39:03.556516 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:03.556351    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" event={"ID":"0c6cec58-b8d9-4856-9446-811407a747ab","Type":"ContainerStarted","Data":"01a5b69e8c051a84622bfd5f598c65f6ed739617d900941ec81338e51762888e"}
Nov 02 17:39:03.557765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:03.557742    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 17:39:03.602514 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:03.602466    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podStartSLOduration=1.6024508320000002 podStartE2EDuration="1.602450832s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:03.600631909 +0000 UTC m=+560.091695070" watchObservedRunningTime="2026-11-02 17:39:03.602450832 +0000 UTC m=+560.093514026"
Nov 02 17:39:04.054335 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:04.054300    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:04.500382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:04.500342    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:05.257044 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:05.257009    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:07.233904 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.233865    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"]
Nov 02 17:39:07.237675 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.237628    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.254853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.254806    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"]
Nov 02 17:39:07.263776 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.263748    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"simulator-w5-863b87-kserve-self-signed-certs\""
Nov 02 17:39:07.263989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.263803    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"kube-root-ca.crt\""
Nov 02 17:39:07.263989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.263867    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"openshift-service-ca.crt\""
Nov 02 17:39:07.266896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.266876    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"default-dockercfg-h47km\""
Nov 02 17:39:07.310802 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.310769    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-tmp-dir\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.311005 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.310850    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-home\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.311082 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.311012    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-dshm\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.311148 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.311100    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d3e197-83a0-4793-8593-0b47b55a9fc4-tls-certs\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.311190 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.311148    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-86tf4\" (UniqueName: \"kubernetes.io/projected/76d3e197-83a0-4793-8593-0b47b55a9fc4-kube-api-access-86tf4\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.311241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.311206    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-model-cache\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.412839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.412420    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-tmp-dir\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.412839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.412499    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-home\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.412839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.412609    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-dshm\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.412839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.412668    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d3e197-83a0-4793-8593-0b47b55a9fc4-tls-certs\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.412839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.412693    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-86tf4\" (UniqueName: \"kubernetes.io/projected/76d3e197-83a0-4793-8593-0b47b55a9fc4-kube-api-access-86tf4\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.412839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.412719    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-model-cache\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.413306 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.413228    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-model-cache\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.416836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.413469    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-tmp-dir\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.416836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.413704    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-home\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.418756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.417551    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-dshm\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.432869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.421142    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d3e197-83a0-4793-8593-0b47b55a9fc4-tls-certs\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.445835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.440686    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-86tf4\" (UniqueName: \"kubernetes.io/projected/76d3e197-83a0-4793-8593-0b47b55a9fc4-kube-api-access-86tf4\") pod \"simulator-w5-863b87-kserve-bd54699cd-7g2mv\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") " pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.518067 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.517735    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:07.551836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.551032    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:07.918301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:07.918263    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"]
Nov 02 17:39:07.919726 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:39:07.919702    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod76d3e197_83a0_4793_8593_0b47b55a9fc4.slice/crio-8dc93a5b3e6f1b88f4d492ce6335a49a9df439e42177e1dfe5ffc1e7ea518989 WatchSource:0}: Error finding container 8dc93a5b3e6f1b88f4d492ce6335a49a9df439e42177e1dfe5ffc1e7ea518989: Status 404 returned error can't find the container with id 8dc93a5b3e6f1b88f4d492ce6335a49a9df439e42177e1dfe5ffc1e7ea518989
Nov 02 17:39:08.644760 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:08.644726    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" event={"ID":"76d3e197-83a0-4793-8593-0b47b55a9fc4","Type":"ContainerStarted","Data":"20ff7d9e053f3451dc21448f1c411ba7fec1a711af2ca0d9541826d17ea0d8cb"}
Nov 02 17:39:08.644760 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:08.644765    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" event={"ID":"76d3e197-83a0-4793-8593-0b47b55a9fc4","Type":"ContainerStarted","Data":"8dc93a5b3e6f1b88f4d492ce6335a49a9df439e42177e1dfe5ffc1e7ea518989"}
Nov 02 17:39:08.693158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:08.693107    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podStartSLOduration=1.69309173 podStartE2EDuration="1.69309173s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:08.691968312 +0000 UTC m=+565.183031502" watchObservedRunningTime="2026-11-02 17:39:08.69309173 +0000 UTC m=+565.184154891"
Nov 02 17:39:08.849276 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:08.849219    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:09.552264 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:09.552228    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:10.532964 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:10.532908    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:12.259237 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:12.259199    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:13.054216 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:13.054183    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:13.069199 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:13.069168    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:13.645403 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:13.645362    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:13.684691 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:13.684660    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:39:14.034639 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:14.034602    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:15.776739 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:15.776679    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:15.927583 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:15.927541    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:16.164013 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:39:16.163930    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:39:17.551917 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:17.551881    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:17.565452 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:17.565424    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:17.744684 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:17.744653    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:39:17.745522 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:17.745493    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:18.487376 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:18.487341    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:19.546906 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:19.546865    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:20.071382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.071344    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"]
Nov 02 17:39:20.075080 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.075055    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.104728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.104702    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"premium-w1-120394-kserve-self-signed-certs\""
Nov 02 17:39:20.108637 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.108610    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"]
Nov 02 17:39:20.169470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.169433    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:20.203661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.203630    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-tmp-dir\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.203661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.203665    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr7dd\" (UniqueName: \"kubernetes.io/projected/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-kube-api-access-lr7dd\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.203942 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.203791    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-tls-certs\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.203942 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.203854    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-dshm\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.204053 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.203989    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-model-cache\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.204145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.204129    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-home\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.305277 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.305237    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-dshm\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.305488 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.305335    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-model-cache\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.305488 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.305443    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-home\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.305488 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.305484    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-tmp-dir\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.305661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.305507    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr7dd\" (UniqueName: \"kubernetes.io/projected/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-kube-api-access-lr7dd\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.305661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.305589    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-tls-certs\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.305965 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.305905    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-tmp-dir\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.306095 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.305964    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-home\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.306095 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.306026    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-model-cache\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.308098 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.308076    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-dshm\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.308579 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.308558    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-tls-certs\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.332112 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.332041    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr7dd\" (UniqueName: \"kubernetes.io/projected/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-kube-api-access-lr7dd\") pod \"premium-w1-120394-kserve-669cf87f88-rmj4s\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") " pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.395336 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.395291    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:20.722140 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.722102    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"]
Nov 02 17:39:20.729022 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:39:20.728971    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod5835a6e8_f2ee_4dd4_9afe_ad0f7a013e34.slice/crio-08103e3b7cf6a358f4d2a88299f447d12720c89aa2456f9bec55e46dc101805e WatchSource:0}: Error finding container 08103e3b7cf6a358f4d2a88299f447d12720c89aa2456f9bec55e46dc101805e: Status 404 returned error can't find the container with id 08103e3b7cf6a358f4d2a88299f447d12720c89aa2456f9bec55e46dc101805e
Nov 02 17:39:20.754510 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:20.754473    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" event={"ID":"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34","Type":"ContainerStarted","Data":"08103e3b7cf6a358f4d2a88299f447d12720c89aa2456f9bec55e46dc101805e"}
Nov 02 17:39:21.350550 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:21.350498    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:21.761904 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:21.761862    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" event={"ID":"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34","Type":"ContainerStarted","Data":"1cbb553419640bc05c0058aa78a8963cbe07f8e1708345f3d87e0220045f11b2"}
Nov 02 17:39:21.807097 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:21.807030    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podStartSLOduration=1.8070156929999999 podStartE2EDuration="1.807015693s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:21.804242935 +0000 UTC m=+578.295306096" watchObservedRunningTime="2026-11-02 17:39:21.807015693 +0000 UTC m=+578.298078857"
Nov 02 17:39:22.395682 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:22.395647    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:23.096744 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:23.096709    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:24.891417 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:24.891368    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:26.255773 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:26.255677    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:26.799963 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:26.799921    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:28.579295 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:28.579259    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:30.340124 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:30.340081    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:30.396385 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:30.396348    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:30.410445 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:30.410418    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:30.835471 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:30.835430    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:39:32.194149 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:32.194112    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:33.191604 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:33.191559    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:34.077144 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:34.077100    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:35.957179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:35.957128    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:37.102842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:37.102742    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:37.742414 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:37.742381    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:39.456233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:39.456173    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:41.192284 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:41.192211    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:43.053190 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:43.053151    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:44.421001 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.420961    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"]
Nov 02 17:39:44.428846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.428792    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.442803 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.442776    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"]
Nov 02 17:39:44.452515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.452493    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"premium-w3-5eedab-kserve-self-signed-certs\""
Nov 02 17:39:44.495903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.495873    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f563607b-3be4-4194-aa10-81e8bf2cbf6e-tls-certs\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.496075 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.496054    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-tmp-dir\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.496118 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.496107    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-home\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.496219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.496203    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-69cwr\" (UniqueName: \"kubernetes.io/projected/f563607b-3be4-4194-aa10-81e8bf2cbf6e-kube-api-access-69cwr\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.496263 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.496252    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-dshm\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.496310 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.496299    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-model-cache\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.501186 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.501154    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:44.597627 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.597591    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-model-cache\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.599178 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.598426    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-model-cache\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.599400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.599374    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f563607b-3be4-4194-aa10-81e8bf2cbf6e-tls-certs\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.599714 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.599694    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-tmp-dir\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.599898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.599872    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-home\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.600172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.599927    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-69cwr\" (UniqueName: \"kubernetes.io/projected/f563607b-3be4-4194-aa10-81e8bf2cbf6e-kube-api-access-69cwr\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.600172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.599987    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-dshm\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.605146 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.602913    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-tmp-dir\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.607517 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.607159    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-home\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.609304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.609279    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-dshm\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.623628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.623583    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f563607b-3be4-4194-aa10-81e8bf2cbf6e-tls-certs\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.634886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.634847    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-69cwr\" (UniqueName: \"kubernetes.io/projected/f563607b-3be4-4194-aa10-81e8bf2cbf6e-kube-api-access-69cwr\") pod \"premium-w3-5eedab-kserve-786db78999-f8lbq\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") " pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.745395 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.745326    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:44.858888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:44.858852    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:45.057419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:45.057393    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"]
Nov 02 17:39:45.059198 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:39:45.059173    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-podf563607b_3be4_4194_aa10_81e8bf2cbf6e.slice/crio-abb99a7cd1f8a60cc21779e528634a40031055d969f794a8c36d0090bcb1ff3e WatchSource:0}: Error finding container abb99a7cd1f8a60cc21779e528634a40031055d969f794a8c36d0090bcb1ff3e: Status 404 returned error can't find the container with id abb99a7cd1f8a60cc21779e528634a40031055d969f794a8c36d0090bcb1ff3e
Nov 02 17:39:45.919866 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:45.919788    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" event={"ID":"f563607b-3be4-4194-aa10-81e8bf2cbf6e","Type":"ContainerStarted","Data":"a3d1aa796ad6acbcdaf08b6699b2eaf2cfb7eaf3f65b1073198205e1ca33cd12"}
Nov 02 17:39:45.920304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:45.919873    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" event={"ID":"f563607b-3be4-4194-aa10-81e8bf2cbf6e","Type":"ContainerStarted","Data":"abb99a7cd1f8a60cc21779e528634a40031055d969f794a8c36d0090bcb1ff3e"}
Nov 02 17:39:45.963363 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:45.963312    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podStartSLOduration=1.9632984850000001 podStartE2EDuration="1.963298485s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:39:45.961662444 +0000 UTC m=+602.452725625" watchObservedRunningTime="2026-11-02 17:39:45.963298485 +0000 UTC m=+602.454361682"
Nov 02 17:39:46.166391 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:39:46.166350    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:39:46.679154 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:46.677133    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:46.745843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:46.745788    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:47.213163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:47.213124    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:48.450503 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:48.450462    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:50.283059 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:50.283025    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:50.521885 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:50.521822    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:52.046547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:52.046505    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:54.084690 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:54.084591    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:54.259135 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:54.259102    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:54.748975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:54.748879    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:54.766162 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:54.766132    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:54.988498 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:54.988470    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:39:55.947446 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:55.947401    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:57.760385 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:57.760353    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:58.962469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:58.962433    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"]
Nov 02 17:39:58.965345 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:58.965328    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.004977 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.004946    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"kube-root-ca.crt\""
Nov 02 17:39:59.005499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.005474    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"openshift-service-ca.crt\""
Nov 02 17:39:59.005499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.005492    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"default-dockercfg-8dv9q\""
Nov 02 17:39:59.005740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.005630    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"premium-w0-069fa5-kserve-self-signed-certs\""
Nov 02 17:39:59.009111 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.009086    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"]
Nov 02 17:39:59.044690 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.044657    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:39:59.080267 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.080224    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-home\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.080460 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.080270    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-dshm\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.080460 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.080417    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-model-cache\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.080598 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.080512    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-tmp-dir\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.080598 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.080579    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wxhm9\" (UniqueName: \"kubernetes.io/projected/9e85cf3f-eb7a-452b-b716-26df5b391065-kube-api-access-wxhm9\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.080707 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.080656    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e85cf3f-eb7a-452b-b716-26df5b391065-tls-certs\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.182271 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.182221    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wxhm9\" (UniqueName: \"kubernetes.io/projected/9e85cf3f-eb7a-452b-b716-26df5b391065-kube-api-access-wxhm9\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.182477 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.182293    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e85cf3f-eb7a-452b-b716-26df5b391065-tls-certs\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.182477 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.182345    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-home\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.182477 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.182382    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-dshm\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.182477 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.182427    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-model-cache\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.182696 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.182505    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-tmp-dir\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.182938 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.182901    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-home\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.183084 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.182994    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-model-cache\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.183084 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.183019    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-tmp-dir\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.185011 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.184983    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-dshm\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.185659 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.185639    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e85cf3f-eb7a-452b-b716-26df5b391065-tls-certs\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.207018 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.206986    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wxhm9\" (UniqueName: \"kubernetes.io/projected/9e85cf3f-eb7a-452b-b716-26df5b391065-kube-api-access-wxhm9\") pod \"premium-w0-069fa5-kserve-699ddf764c-xmspr\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") " pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.277016 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.276929    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:39:59.585261 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.585221    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"]
Nov 02 17:39:59.616049 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:39:59.616018    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:00.006550 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:00.006517    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" event={"ID":"9e85cf3f-eb7a-452b-b716-26df5b391065","Type":"ContainerStarted","Data":"d88d0bddc4d93ed42f5f2ffb943fc376a1de597ed4368f492c206bca841a6ef3"}
Nov 02 17:40:00.006550 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:00.006556    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" event={"ID":"9e85cf3f-eb7a-452b-b716-26df5b391065","Type":"ContainerStarted","Data":"6c6128144aa620bfae7630e97fd32114706f3f724b42af7faa4cb7a7c554ab1e"}
Nov 02 17:40:00.051737 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:00.051688    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podStartSLOduration=2.051675115 podStartE2EDuration="2.051675115s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:39:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:00.049223701 +0000 UTC m=+616.540286861" watchObservedRunningTime="2026-11-02 17:40:00.051675115 +0000 UTC m=+616.542738276"
Nov 02 17:40:00.277542 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:00.277467    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:40:01.464609 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:01.464573    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:03.269320 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:03.269286    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:05.059044 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:05.059003    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:06.845757 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:06.845720    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:08.598431 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:08.598364    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:09.277318 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:09.277282    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:40:09.290208 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:09.290180    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:40:10.081181 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:10.081152    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:40:10.369373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:10.369284    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:12.231839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:12.231477    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:14.138326 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:14.138288    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:15.954972 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:15.954930    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:16.015859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.015800    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"]
Nov 02 17:40:16.020541 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.020517    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.051218 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.051188    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-863b87-openshift-default-dockercfg-5mwm4\""
Nov 02 17:40:16.055224 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.055195    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"]
Nov 02 17:40:16.120330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.120296    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.120330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.120331    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4w5gc\" (UniqueName: \"kubernetes.io/projected/22b39708-251c-44ad-9b48-2f3c71519131-kube-api-access-4w5gc\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.120528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.120413    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.120528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.120439    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.120528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.120473    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.120528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.120524    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b39708-251c-44ad-9b48-2f3c71519131-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.120658 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.120568    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b39708-251c-44ad-9b48-2f3c71519131-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.120658 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.120634    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.120658 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.120649    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b39708-251c-44ad-9b48-2f3c71519131-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.165220 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:40:16.165187    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:40:16.221906 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.221803    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.221906 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.221866    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b39708-251c-44ad-9b48-2f3c71519131-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.222136 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.221967    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.222136 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.221994    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4w5gc\" (UniqueName: \"kubernetes.io/projected/22b39708-251c-44ad-9b48-2f3c71519131-kube-api-access-4w5gc\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.222136 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.222115    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.222325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.222168    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.222325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.222199    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.222325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.222238    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.222325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.222256    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b39708-251c-44ad-9b48-2f3c71519131-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.222325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.222304    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b39708-251c-44ad-9b48-2f3c71519131-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.222605 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.222512    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.222747 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.222667    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.222930 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.222916    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.223404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.223375    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b39708-251c-44ad-9b48-2f3c71519131-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.224490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.224469    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.224806 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.224784    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b39708-251c-44ad-9b48-2f3c71519131-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.256713 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.256689    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4w5gc\" (UniqueName: \"kubernetes.io/projected/22b39708-251c-44ad-9b48-2f3c71519131-kube-api-access-4w5gc\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.257770 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.257752    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b39708-251c-44ad-9b48-2f3c71519131-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.335343 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.335312    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:16.678872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.678774    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"]
Nov 02 17:40:16.683383 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:40:16.683355    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b39708_251c_44ad_9b48_2f3c71519131.slice/crio-c3ee98a2f153d0f496edfe95bba4b2fbf19a84c33456c0ccc6eb86e4eac9cc3a WatchSource:0}: Error finding container c3ee98a2f153d0f496edfe95bba4b2fbf19a84c33456c0ccc6eb86e4eac9cc3a: Status 404 returned error can't find the container with id c3ee98a2f153d0f496edfe95bba4b2fbf19a84c33456c0ccc6eb86e4eac9cc3a
Nov 02 17:40:16.686107 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.686072    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:16.686217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.686157    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:16.686217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:16.686203    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:17.126165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:17.125999    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj" event={"ID":"22b39708-251c-44ad-9b48-2f3c71519131","Type":"ContainerStarted","Data":"eb26dc382b0920d691696f96b4ff0e70ad2fad6dae676cf773938a373d8aa28d"}
Nov 02 17:40:17.126165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:17.126088    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj" event={"ID":"22b39708-251c-44ad-9b48-2f3c71519131","Type":"ContainerStarted","Data":"c3ee98a2f153d0f496edfe95bba4b2fbf19a84c33456c0ccc6eb86e4eac9cc3a"}
Nov 02 17:40:17.185208 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:17.185158    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj" podStartSLOduration=2.185142116 podStartE2EDuration="2.185142116s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:17.184324117 +0000 UTC m=+633.675387279" watchObservedRunningTime="2026-11-02 17:40:17.185142116 +0000 UTC m=+633.676205278"
Nov 02 17:40:17.335890 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:17.335856    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:17.337550 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:17.337520    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:40:17.337682 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:17.337582    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj" podUID="22b39708-251c-44ad-9b48-2f3c71519131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 17:40:17.763941 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:17.763462    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:18.337860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:18.336478    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:40:18.337860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:18.336561    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj" podUID="22b39708-251c-44ad-9b48-2f3c71519131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 17:40:19.336599 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:19.336566    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:40:19.336786 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:19.336646    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj" podUID="22b39708-251c-44ad-9b48-2f3c71519131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 17:40:19.553664 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:19.553628    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:20.335981 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:20.335945    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:40:20.336204 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:20.336003    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj" podUID="22b39708-251c-44ad-9b48-2f3c71519131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 17:40:21.336608 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:21.336570    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:40:21.337110 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:21.336624    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj" podUID="22b39708-251c-44ad-9b48-2f3c71519131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 17:40:21.362017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:21.361979    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:22.335889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:22.335842    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:40:22.336101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:22.335913    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj" podUID="22b39708-251c-44ad-9b48-2f3c71519131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 17:40:23.284860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:23.283357    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:23.336205 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:23.336167    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused" start-of-body=
Nov 02 17:40:23.336463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:23.336235    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj" podUID="22b39708-251c-44ad-9b48-2f3c71519131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": dial tcp 10.134.0.53:15021: connect: connection refused"
Nov 02 17:40:24.340759 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:24.340725    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:24.341211 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:24.341191    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:24.341880 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:24.341856    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:40:25.368627 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:25.368578    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:27.256090 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.256058    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:27.440471 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.440435    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"]
Nov 02 17:40:27.445295 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.445276    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"
Nov 02 17:40:27.463368 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.463341    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"]
Nov 02 17:40:27.472624 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.472588    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:40:27.472624 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.472598    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:40:27.592252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.592152    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5dd46c7-cs5pr\" (UID: \"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"
Nov 02 17:40:27.592252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.592196    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sjm59\" (UniqueName: \"kubernetes.io/projected/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-kube-api-access-sjm59\") pod \"maas-api-revoke-sub-e5dd46c7-cs5pr\" (UID: \"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"
Nov 02 17:40:27.592478 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.592296    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5dd46c7-cs5pr\" (UID: \"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"
Nov 02 17:40:27.693751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.693710    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5dd46c7-cs5pr\" (UID: \"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"
Nov 02 17:40:27.693751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.693751    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sjm59\" (UniqueName: \"kubernetes.io/projected/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-kube-api-access-sjm59\") pod \"maas-api-revoke-sub-e5dd46c7-cs5pr\" (UID: \"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"
Nov 02 17:40:27.694020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.693798    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5dd46c7-cs5pr\" (UID: \"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"
Nov 02 17:40:27.694621 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.694550    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e5dd46c7-cs5pr\" (UID: \"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"
Nov 02 17:40:27.696272 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.696252    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e5dd46c7-cs5pr\" (UID: \"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"
Nov 02 17:40:27.717671 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.717648    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sjm59\" (UniqueName: \"kubernetes.io/projected/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-kube-api-access-sjm59\") pod \"maas-api-revoke-sub-e5dd46c7-cs5pr\" (UID: \"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"
Nov 02 17:40:27.759609 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:27.759583    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"
Nov 02 17:40:28.074950 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:28.074921    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"]
Nov 02 17:40:28.075665 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:40:28.075639    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3f7e4b49_d0a8_4409_899b_7fd9b9e51d24.slice/crio-bb6e591abae13085d322deb69267f91acd68fd688024673964dab4e5c5ca8ed4 WatchSource:0}: Error finding container bb6e591abae13085d322deb69267f91acd68fd688024673964dab4e5c5ca8ed4: Status 404 returned error can't find the container with id bb6e591abae13085d322deb69267f91acd68fd688024673964dab4e5c5ca8ed4
Nov 02 17:40:28.202558 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:28.202521    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr" event={"ID":"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24","Type":"ContainerStarted","Data":"bb6e591abae13085d322deb69267f91acd68fd688024673964dab4e5c5ca8ed4"}
Nov 02 17:40:29.063071 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:29.062995    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:30.829993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:30.829898    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:31.234450 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:31.234236    4456 generic.go:356] "Generic (PLEG): container finished" podID="3f7e4b49-d0a8-4409-899b-7fd9b9e51d24" containerID="b0a3d6acd937a153cbcf8537f2fe0c5fffc4dfdbd6773912c0f04f13722892f1" exitCode=0
Nov 02 17:40:31.234450 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:31.234377    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr" event={"ID":"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24","Type":"ContainerDied","Data":"b0a3d6acd937a153cbcf8537f2fe0c5fffc4dfdbd6773912c0f04f13722892f1"}
Nov 02 17:40:32.670089 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:32.670057    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:32.682064 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:32.682038    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"
Nov 02 17:40:32.817260 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:32.817203    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-maas-api-openshift-ca\") pod \"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24\" (UID: \"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24\") "
Nov 02 17:40:32.817456 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:32.817276    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-maas-api-xks-ca\") pod \"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24\" (UID: \"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24\") "
Nov 02 17:40:32.817456 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:32.817352    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-kube-api-access-sjm59\" (UniqueName: \"kubernetes.io/projected/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-kube-api-access-sjm59\") pod \"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24\" (UID: \"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24\") "
Nov 02 17:40:32.818057 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:32.818029    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-maas-api-openshift-ca" pod "3f7e4b49-d0a8-4409-899b-7fd9b9e51d24" (UID: "3f7e4b49-d0a8-4409-899b-7fd9b9e51d24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:40:32.820280 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:32.820254    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-maas-api-xks-ca" pod "3f7e4b49-d0a8-4409-899b-7fd9b9e51d24" (UID: "3f7e4b49-d0a8-4409-899b-7fd9b9e51d24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:40:32.820616 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:32.820596    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-kube-api-access-sjm59" pod "3f7e4b49-d0a8-4409-899b-7fd9b9e51d24" (UID: "3f7e4b49-d0a8-4409-899b-7fd9b9e51d24"). InnerVolumeSpecName "kube-api-access-sjm59". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:40:32.918715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:32.918636    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:40:32.918715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:32.918666    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:40:32.918715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:32.918677    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sjm59\" (UniqueName: \"kubernetes.io/projected/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24-kube-api-access-sjm59\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:40:33.254018 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:33.253984    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"
Nov 02 17:40:33.254199 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:33.254009    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr" event={"ID":"3f7e4b49-d0a8-4409-899b-7fd9b9e51d24","Type":"ContainerDied","Data":"bb6e591abae13085d322deb69267f91acd68fd688024673964dab4e5c5ca8ed4"}
Nov 02 17:40:33.254199 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:33.254067    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bb6e591abae13085d322deb69267f91acd68fd688024673964dab4e5c5ca8ed4"
Nov 02 17:40:33.333649 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:33.333618    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:34.671457 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:34.671419    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:36.537346 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:36.537309    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:38.372420 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:38.372385    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:40.248919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:40.248883    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:42.058060 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.056214    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:42.402454 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.402378    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"]
Nov 02 17:40:42.402884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.402871    4456 cpu_manager.go:414] "containerMap: removing container" podUID="3f7e4b49-d0a8-4409-899b-7fd9b9e51d24" containerName="revoke-keys"
Nov 02 17:40:42.402938 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.402886    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3f7e4b49-d0a8-4409-899b-7fd9b9e51d24" containerName="revoke-keys"
Nov 02 17:40:42.403005 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.402996    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="3f7e4b49-d0a8-4409-899b-7fd9b9e51d24" containerName="revoke-keys"
Nov 02 17:40:42.406319 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.406300    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"
Nov 02 17:40:42.424980 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.424947    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"]
Nov 02 17:40:42.433586 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.433563    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:40:42.433586 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.433560    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:40:42.534446 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.534415    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dcd3478-d9ktv\" (UID: \"06ef6b4f-c98a-4655-8b92-6d6930eec8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"
Nov 02 17:40:42.534619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.534571    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dcd3478-d9ktv\" (UID: \"06ef6b4f-c98a-4655-8b92-6d6930eec8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"
Nov 02 17:40:42.534619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.534602    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9dtw\" (UniqueName: \"kubernetes.io/projected/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-kube-api-access-p9dtw\") pod \"maas-api-revoke-sub-2dcd3478-d9ktv\" (UID: \"06ef6b4f-c98a-4655-8b92-6d6930eec8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"
Nov 02 17:40:42.636332 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.636287    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dcd3478-d9ktv\" (UID: \"06ef6b4f-c98a-4655-8b92-6d6930eec8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"
Nov 02 17:40:42.636332 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.636338    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9dtw\" (UniqueName: \"kubernetes.io/projected/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-kube-api-access-p9dtw\") pod \"maas-api-revoke-sub-2dcd3478-d9ktv\" (UID: \"06ef6b4f-c98a-4655-8b92-6d6930eec8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"
Nov 02 17:40:42.636595 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.636408    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dcd3478-d9ktv\" (UID: \"06ef6b4f-c98a-4655-8b92-6d6930eec8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"
Nov 02 17:40:42.637152 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.637133    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2dcd3478-d9ktv\" (UID: \"06ef6b4f-c98a-4655-8b92-6d6930eec8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"
Nov 02 17:40:42.639095 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.639065    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2dcd3478-d9ktv\" (UID: \"06ef6b4f-c98a-4655-8b92-6d6930eec8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"
Nov 02 17:40:42.660318 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.660261    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9dtw\" (UniqueName: \"kubernetes.io/projected/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-kube-api-access-p9dtw\") pod \"maas-api-revoke-sub-2dcd3478-d9ktv\" (UID: \"06ef6b4f-c98a-4655-8b92-6d6930eec8a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"
Nov 02 17:40:42.719320 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:42.719285    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"
Nov 02 17:40:43.027731 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:43.027708    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"]
Nov 02 17:40:43.030179 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:40:43.030154    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod06ef6b4f_c98a_4655_8b92_6d6930eec8a1.slice/crio-0d5488143456c21a836ea1fb5de10db423d68078ee86cbf4ba4e05992ddb157d WatchSource:0}: Error finding container 0d5488143456c21a836ea1fb5de10db423d68078ee86cbf4ba4e05992ddb157d: Status 404 returned error can't find the container with id 0d5488143456c21a836ea1fb5de10db423d68078ee86cbf4ba4e05992ddb157d
Nov 02 17:40:43.326980 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:43.326945    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv" event={"ID":"06ef6b4f-c98a-4655-8b92-6d6930eec8a1","Type":"ContainerStarted","Data":"8c6f0412d220391d3a2f9412595ab8c617a940b75a1a6eddce8ad00121b6934c"}
Nov 02 17:40:43.327449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:43.327011    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv" event={"ID":"06ef6b4f-c98a-4655-8b92-6d6930eec8a1","Type":"ContainerStarted","Data":"0d5488143456c21a836ea1fb5de10db423d68078ee86cbf4ba4e05992ddb157d"}
Nov 02 17:40:43.385360 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:43.385293    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv" podStartSLOduration=1.385274495 podStartE2EDuration="1.385274495s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:43.368953287 +0000 UTC m=+659.860016458" watchObservedRunningTime="2026-11-02 17:40:43.385274495 +0000 UTC m=+659.876337658"
Nov 02 17:40:43.856981 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:43.856949    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:44.334758 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:44.334721    4456 generic.go:356] "Generic (PLEG): container finished" podID="06ef6b4f-c98a-4655-8b92-6d6930eec8a1" containerID="8c6f0412d220391d3a2f9412595ab8c617a940b75a1a6eddce8ad00121b6934c" exitCode=0
Nov 02 17:40:44.335190 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:44.334780    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv" event={"ID":"06ef6b4f-c98a-4655-8b92-6d6930eec8a1","Type":"ContainerDied","Data":"8c6f0412d220391d3a2f9412595ab8c617a940b75a1a6eddce8ad00121b6934c"}
Nov 02 17:40:45.496574 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.496545    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"
Nov 02 17:40:45.567687 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.567655    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-maas-api-xks-ca\") pod \"06ef6b4f-c98a-4655-8b92-6d6930eec8a1\" (UID: \"06ef6b4f-c98a-4655-8b92-6d6930eec8a1\") "
Nov 02 17:40:45.567888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.567726    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-kube-api-access-p9dtw\" (UniqueName: \"kubernetes.io/projected/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-kube-api-access-p9dtw\") pod \"06ef6b4f-c98a-4655-8b92-6d6930eec8a1\" (UID: \"06ef6b4f-c98a-4655-8b92-6d6930eec8a1\") "
Nov 02 17:40:45.567977 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.567894    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-maas-api-openshift-ca\") pod \"06ef6b4f-c98a-4655-8b92-6d6930eec8a1\" (UID: \"06ef6b4f-c98a-4655-8b92-6d6930eec8a1\") "
Nov 02 17:40:45.568288 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.568249    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-maas-api-openshift-ca" pod "06ef6b4f-c98a-4655-8b92-6d6930eec8a1" (UID: "06ef6b4f-c98a-4655-8b92-6d6930eec8a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:40:45.568404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.568363    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:40:45.570544 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.570520    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-maas-api-xks-ca" pod "06ef6b4f-c98a-4655-8b92-6d6930eec8a1" (UID: "06ef6b4f-c98a-4655-8b92-6d6930eec8a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:40:45.570652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.570568    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-kube-api-access-p9dtw" pod "06ef6b4f-c98a-4655-8b92-6d6930eec8a1" (UID: "06ef6b4f-c98a-4655-8b92-6d6930eec8a1"). InnerVolumeSpecName "kube-api-access-p9dtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:40:45.669315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.669280    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:40:45.669315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.669315    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9dtw\" (UniqueName: \"kubernetes.io/projected/06ef6b4f-c98a-4655-8b92-6d6930eec8a1-kube-api-access-p9dtw\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:40:45.764686 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.764647    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:45.908711 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.908629    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"]
Nov 02 17:40:45.909214 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.909196    4456 cpu_manager.go:414] "containerMap: removing container" podUID="06ef6b4f-c98a-4655-8b92-6d6930eec8a1" containerName="revoke-keys"
Nov 02 17:40:45.909264 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.909216    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06ef6b4f-c98a-4655-8b92-6d6930eec8a1" containerName="revoke-keys"
Nov 02 17:40:45.909322 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.909314    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="06ef6b4f-c98a-4655-8b92-6d6930eec8a1" containerName="revoke-keys"
Nov 02 17:40:45.912631 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.912615    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:45.947773 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.947733    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5eedab-openshift-default-dockercfg-57c6m\""
Nov 02 17:40:45.950599 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.950570    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"]
Nov 02 17:40:45.972134 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.972111    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:45.972258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.972147    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:45.972308 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.972277    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zq8ts\" (UniqueName: \"kubernetes.io/projected/f56cfa88-cc79-4d08-9111-b47d940533ba-kube-api-access-zq8ts\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:45.972354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.972305    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f56cfa88-cc79-4d08-9111-b47d940533ba-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:45.972354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.972335    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:45.972470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.972451    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:45.972556 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.972512    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:45.972599 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.972556    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:45.972599 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:45.972577    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.074157 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.074126    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.074348 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.074186    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.074348 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.074210    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.074348 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.074234    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.074348 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.074254    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.074348 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.074280    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.074628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.074393    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zq8ts\" (UniqueName: \"kubernetes.io/projected/f56cfa88-cc79-4d08-9111-b47d940533ba-kube-api-access-zq8ts\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.074628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.074432    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f56cfa88-cc79-4d08-9111-b47d940533ba-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.074628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.074471    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.074772 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.074651    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.074772 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.074738    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.074910 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.074771    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.074970 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.074954    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.075265 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.075239    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f56cfa88-cc79-4d08-9111-b47d940533ba-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.076734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.076711    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.077026 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.077009    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.102246 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.102209    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.102404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.102381    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zq8ts\" (UniqueName: \"kubernetes.io/projected/f56cfa88-cc79-4d08-9111-b47d940533ba-kube-api-access-zq8ts\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.165915 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:40:46.165828    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:40:46.230189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.230162    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:46.303887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.302017    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"]
Nov 02 17:40:46.315673 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.314113    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.335049 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.329253    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"]
Nov 02 17:40:46.335049 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.331013    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2899ab-openshift-default-dockercfg-t6vss\""
Nov 02 17:40:46.352724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.352688    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv" event={"ID":"06ef6b4f-c98a-4655-8b92-6d6930eec8a1","Type":"ContainerDied","Data":"0d5488143456c21a836ea1fb5de10db423d68078ee86cbf4ba4e05992ddb157d"}
Nov 02 17:40:46.352903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.352752    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0d5488143456c21a836ea1fb5de10db423d68078ee86cbf4ba4e05992ddb157d"
Nov 02 17:40:46.352903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.352862    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"
Nov 02 17:40:46.377221 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.377190    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.377373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.377291    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.377373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.377343    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.377461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.377384    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tsncm\" (UniqueName: \"kubernetes.io/projected/e5167f1d-3718-4c96-bf19-1cdc1ab73207-kube-api-access-tsncm\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.377461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.377445    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.377536 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.377518    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.377568 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.377544    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.377568 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.377562    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.377640 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.377627    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.478662 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.478622    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.478889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.478734    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.478889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.478798    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.479040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.478902    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.479751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.479125    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.479751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.479146    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tsncm\" (UniqueName: \"kubernetes.io/projected/e5167f1d-3718-4c96-bf19-1cdc1ab73207-kube-api-access-tsncm\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.479751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.479247    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.479751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.479261    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.479751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.479299    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.479751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.479334    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.479751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.479360    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.479751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.479638    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.479751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.479696    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.480310 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.479906    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.481946 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.481916    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.482279 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.482262    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.507792 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.507763    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.508163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.508140    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tsncm\" (UniqueName: \"kubernetes.io/projected/e5167f1d-3718-4c96-bf19-1cdc1ab73207-kube-api-access-tsncm\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.562461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.562433    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"]
Nov 02 17:40:46.564005 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:40:46.563963    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf56cfa88_cc79_4d08_9111_b47d940533ba.slice/crio-22c6aba7e42ad31aa390d0dac09323af68b95770b49f8b3199b2570582875c02 WatchSource:0}: Error finding container 22c6aba7e42ad31aa390d0dac09323af68b95770b49f8b3199b2570582875c02: Status 404 returned error can't find the container with id 22c6aba7e42ad31aa390d0dac09323af68b95770b49f8b3199b2570582875c02
Nov 02 17:40:46.566656 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.566618    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:46.566760 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.566701    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:46.566760 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.566743    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:46.650222 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.650196    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:46.997883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:46.997854    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"]
Nov 02 17:40:46.999731 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:40:46.999701    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode5167f1d_3718_4c96_bf19_1cdc1ab73207.slice/crio-2385f09846039394e104b7d8241a2008a6cfba0db6f192dd2c1616316880d45f WatchSource:0}: Error finding container 2385f09846039394e104b7d8241a2008a6cfba0db6f192dd2c1616316880d45f: Status 404 returned error can't find the container with id 2385f09846039394e104b7d8241a2008a6cfba0db6f192dd2c1616316880d45f
Nov 02 17:40:47.002424 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:47.002346    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:47.002424 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:47.002412    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:47.002663 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:47.002451    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:40:47.361738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:47.361626    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" event={"ID":"e5167f1d-3718-4c96-bf19-1cdc1ab73207","Type":"ContainerStarted","Data":"aaaa12bbfe6411dd90db578524e8a9702da102f37573e9977362c806a70e72b3"}
Nov 02 17:40:47.361738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:47.361697    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" event={"ID":"e5167f1d-3718-4c96-bf19-1cdc1ab73207","Type":"ContainerStarted","Data":"2385f09846039394e104b7d8241a2008a6cfba0db6f192dd2c1616316880d45f"}
Nov 02 17:40:47.364068 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:47.364035    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" event={"ID":"f56cfa88-cc79-4d08-9111-b47d940533ba","Type":"ContainerStarted","Data":"4db8580f4eb63d66c47f9f317c68868ee1b4d374974981f9e830b2a3f1cea2f5"}
Nov 02 17:40:47.364193 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:47.364092    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" event={"ID":"f56cfa88-cc79-4d08-9111-b47d940533ba","Type":"ContainerStarted","Data":"22c6aba7e42ad31aa390d0dac09323af68b95770b49f8b3199b2570582875c02"}
Nov 02 17:40:47.418871 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:47.412186    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" podStartSLOduration=1.412166545 podStartE2EDuration="1.412166545s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:47.407280257 +0000 UTC m=+663.898343429" watchObservedRunningTime="2026-11-02 17:40:47.412166545 +0000 UTC m=+663.903229707"
Nov 02 17:40:47.457291 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:47.457223    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" podStartSLOduration=2.457205086 podStartE2EDuration="2.457205086s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:45 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:47.453594419 +0000 UTC m=+663.944657618" watchObservedRunningTime="2026-11-02 17:40:47.457205086 +0000 UTC m=+663.948268251"
Nov 02 17:40:47.565573 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:47.565532    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:47.654040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:47.653951    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:48.234249 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:48.234207    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:48.237957 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:48.237926    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 17:40:48.238212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:48.238177    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 17:40:48.653695 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:48.653603    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 02 17:40:48.654322 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:48.654288    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 02 17:40:49.231262 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:49.231225    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 17:40:49.231549 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:49.231518    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 17:40:49.349120 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:49.349084    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:49.650786 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:49.650686    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 02 17:40:49.650786 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:49.650755    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 02 17:40:50.239008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:50.238967    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 17:40:50.239553 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:50.239033    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 17:40:50.651357 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:50.651262    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 02 17:40:50.651657 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:50.651623    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 02 17:40:51.166420 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:51.166382    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:51.236218 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:51.236175    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 17:40:51.236439 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:51.236243    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 17:40:51.657092 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:51.650983    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 02 17:40:51.657092 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:51.651133    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 02 17:40:52.237892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:52.237855    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 17:40:52.238171 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:52.238143    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 17:40:52.651544 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:52.651453    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 02 17:40:52.651544 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:52.651510    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 02 17:40:53.025483 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:53.025411    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:53.231213 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:53.231178    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused" start-of-body=
Nov 02 17:40:53.231511 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:53.231468    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": dial tcp 10.134.0.56:15021: connect: connection refused"
Nov 02 17:40:53.665837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:53.665008    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused" start-of-body=
Nov 02 17:40:53.665837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:53.665104    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": dial tcp 10.134.0.57:15021: connect: connection refused"
Nov 02 17:40:54.236749 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:54.236715    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:54.237271 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:54.237208    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:54.237931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:54.237910    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 17:40:54.730807 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:54.730779    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:54.731869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:54.731844    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:54.733033 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:54.733009    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 17:40:55.070417 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:55.070303    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:56.860734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:56.860698    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:57.153525 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.153449    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"]
Nov 02 17:40:57.158559 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.158541    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"
Nov 02 17:40:57.180195 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.180172    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"]
Nov 02 17:40:57.187717 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.187691    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:40:57.187863 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.187796    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:40:57.267866 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.267793    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3484eb28-e4be-4867-a6e6-aac7908e8124-maas-api-xks-ca\") pod \"maas-api-revoke-sub-367340d0-kvcgj\" (UID: \"3484eb28-e4be-4867-a6e6-aac7908e8124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"
Nov 02 17:40:57.268038 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.267937    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vqg4d\" (UniqueName: \"kubernetes.io/projected/3484eb28-e4be-4867-a6e6-aac7908e8124-kube-api-access-vqg4d\") pod \"maas-api-revoke-sub-367340d0-kvcgj\" (UID: \"3484eb28-e4be-4867-a6e6-aac7908e8124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"
Nov 02 17:40:57.268185 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.268164    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3484eb28-e4be-4867-a6e6-aac7908e8124-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-367340d0-kvcgj\" (UID: \"3484eb28-e4be-4867-a6e6-aac7908e8124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"
Nov 02 17:40:57.369224 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.369181    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vqg4d\" (UniqueName: \"kubernetes.io/projected/3484eb28-e4be-4867-a6e6-aac7908e8124-kube-api-access-vqg4d\") pod \"maas-api-revoke-sub-367340d0-kvcgj\" (UID: \"3484eb28-e4be-4867-a6e6-aac7908e8124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"
Nov 02 17:40:57.369392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.369338    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3484eb28-e4be-4867-a6e6-aac7908e8124-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-367340d0-kvcgj\" (UID: \"3484eb28-e4be-4867-a6e6-aac7908e8124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"
Nov 02 17:40:57.369392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.369368    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3484eb28-e4be-4867-a6e6-aac7908e8124-maas-api-xks-ca\") pod \"maas-api-revoke-sub-367340d0-kvcgj\" (UID: \"3484eb28-e4be-4867-a6e6-aac7908e8124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"
Nov 02 17:40:57.370100 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.370074    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3484eb28-e4be-4867-a6e6-aac7908e8124-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-367340d0-kvcgj\" (UID: \"3484eb28-e4be-4867-a6e6-aac7908e8124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"
Nov 02 17:40:57.371844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.371825    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3484eb28-e4be-4867-a6e6-aac7908e8124-maas-api-xks-ca\") pod \"maas-api-revoke-sub-367340d0-kvcgj\" (UID: \"3484eb28-e4be-4867-a6e6-aac7908e8124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"
Nov 02 17:40:57.393055 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.393027    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vqg4d\" (UniqueName: \"kubernetes.io/projected/3484eb28-e4be-4867-a6e6-aac7908e8124-kube-api-access-vqg4d\") pod \"maas-api-revoke-sub-367340d0-kvcgj\" (UID: \"3484eb28-e4be-4867-a6e6-aac7908e8124\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"
Nov 02 17:40:57.472739 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.472713    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"
Nov 02 17:40:57.797338 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:57.797260    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"]
Nov 02 17:40:57.801974 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:40:57.801946    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3484eb28_e4be_4867_a6e6_aac7908e8124.slice/crio-415b02f3ea0b0e388232d59dc113c7e0cdc15d611a95f8b25db7cbface3e709b WatchSource:0}: Error finding container 415b02f3ea0b0e388232d59dc113c7e0cdc15d611a95f8b25db7cbface3e709b: Status 404 returned error can't find the container with id 415b02f3ea0b0e388232d59dc113c7e0cdc15d611a95f8b25db7cbface3e709b
Nov 02 17:40:58.511238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:58.511203    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj" event={"ID":"3484eb28-e4be-4867-a6e6-aac7908e8124","Type":"ContainerStarted","Data":"99d7fc8b557c0716e7c4eac696f3ab2813ca97ace85eac24a2568f311287af27"}
Nov 02 17:40:58.511659 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:58.511274    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj" event={"ID":"3484eb28-e4be-4867-a6e6-aac7908e8124","Type":"ContainerStarted","Data":"415b02f3ea0b0e388232d59dc113c7e0cdc15d611a95f8b25db7cbface3e709b"}
Nov 02 17:40:58.559634 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:58.559569    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj" podStartSLOduration=1.559550979 podStartE2EDuration="1.559550979s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:40:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:40:58.5553008 +0000 UTC m=+675.046363963" watchObservedRunningTime="2026-11-02 17:40:58.559550979 +0000 UTC m=+675.050614140"
Nov 02 17:40:58.730474 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:58.730422    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:40:59.520504 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:59.520464    4456 generic.go:356] "Generic (PLEG): container finished" podID="3484eb28-e4be-4867-a6e6-aac7908e8124" containerID="99d7fc8b557c0716e7c4eac696f3ab2813ca97ace85eac24a2568f311287af27" exitCode=0
Nov 02 17:40:59.520950 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:40:59.520532    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj" event={"ID":"3484eb28-e4be-4867-a6e6-aac7908e8124","Type":"ContainerDied","Data":"99d7fc8b557c0716e7c4eac696f3ab2813ca97ace85eac24a2568f311287af27"}
Nov 02 17:41:00.165548 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:00.165482    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"]
Nov 02 17:41:00.170806 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:00.170775    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"
Nov 02 17:41:00.192839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:00.192795    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"]
Nov 02 17:41:00.332340 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:00.332301    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f4e882-e05a-4ad5-8cfe-20117be35a46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14b6302-tv6k4\" (UID: \"a0f4e882-e05a-4ad5-8cfe-20117be35a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"
Nov 02 17:41:00.332533 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:00.332343    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fjft8\" (UniqueName: \"kubernetes.io/projected/a0f4e882-e05a-4ad5-8cfe-20117be35a46-kube-api-access-fjft8\") pod \"maas-api-revoke-sub-f14b6302-tv6k4\" (UID: \"a0f4e882-e05a-4ad5-8cfe-20117be35a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"
Nov 02 17:41:00.332609 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:00.332578    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f4e882-e05a-4ad5-8cfe-20117be35a46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14b6302-tv6k4\" (UID: \"a0f4e882-e05a-4ad5-8cfe-20117be35a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"
Nov 02 17:41:00.433763 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:00.433726    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f4e882-e05a-4ad5-8cfe-20117be35a46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14b6302-tv6k4\" (UID: \"a0f4e882-e05a-4ad5-8cfe-20117be35a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"
Nov 02 17:41:00.434895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:00.434860    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fjft8\" (UniqueName: \"kubernetes.io/projected/a0f4e882-e05a-4ad5-8cfe-20117be35a46-kube-api-access-fjft8\") pod \"maas-api-revoke-sub-f14b6302-tv6k4\" (UID: \"a0f4e882-e05a-4ad5-8cfe-20117be35a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"
Nov 02 17:41:00.435435 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:00.434764    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f4e882-e05a-4ad5-8cfe-20117be35a46-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f14b6302-tv6k4\" (UID: \"a0f4e882-e05a-4ad5-8cfe-20117be35a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"
Nov 02 17:41:00.435759 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:00.435736    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f4e882-e05a-4ad5-8cfe-20117be35a46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14b6302-tv6k4\" (UID: \"a0f4e882-e05a-4ad5-8cfe-20117be35a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"
Nov 02 17:41:00.440203 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:00.440183    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f4e882-e05a-4ad5-8cfe-20117be35a46-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f14b6302-tv6k4\" (UID: \"a0f4e882-e05a-4ad5-8cfe-20117be35a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"
Nov 02 17:41:00.466959 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:00.466934    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fjft8\" (UniqueName: \"kubernetes.io/projected/a0f4e882-e05a-4ad5-8cfe-20117be35a46-kube-api-access-fjft8\") pod \"maas-api-revoke-sub-f14b6302-tv6k4\" (UID: \"a0f4e882-e05a-4ad5-8cfe-20117be35a46\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"
Nov 02 17:41:00.503370 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:00.502039    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"
Nov 02 17:41:00.661489 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:00.661370    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:01.011036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.011009    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"
Nov 02 17:41:01.091924 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.091897    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"]
Nov 02 17:41:01.093525 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:41:01.093497    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda0f4e882_e05a_4ad5_8cfe_20117be35a46.slice/crio-fdf2978bafea5d87f88e8bb944cabd1f472a85a75a8e6b5cd115c444ce591cb5 WatchSource:0}: Error finding container fdf2978bafea5d87f88e8bb944cabd1f472a85a75a8e6b5cd115c444ce591cb5: Status 404 returned error can't find the container with id fdf2978bafea5d87f88e8bb944cabd1f472a85a75a8e6b5cd115c444ce591cb5
Nov 02 17:41:01.179940 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.179907    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3484eb28-e4be-4867-a6e6-aac7908e8124-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3484eb28-e4be-4867-a6e6-aac7908e8124-maas-api-xks-ca\") pod \"3484eb28-e4be-4867-a6e6-aac7908e8124\" (UID: \"3484eb28-e4be-4867-a6e6-aac7908e8124\") "
Nov 02 17:41:01.180112 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.180000    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3484eb28-e4be-4867-a6e6-aac7908e8124-kube-api-access-vqg4d\" (UniqueName: \"kubernetes.io/projected/3484eb28-e4be-4867-a6e6-aac7908e8124-kube-api-access-vqg4d\") pod \"3484eb28-e4be-4867-a6e6-aac7908e8124\" (UID: \"3484eb28-e4be-4867-a6e6-aac7908e8124\") "
Nov 02 17:41:01.180112 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.180028    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3484eb28-e4be-4867-a6e6-aac7908e8124-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3484eb28-e4be-4867-a6e6-aac7908e8124-maas-api-openshift-ca\") pod \"3484eb28-e4be-4867-a6e6-aac7908e8124\" (UID: \"3484eb28-e4be-4867-a6e6-aac7908e8124\") "
Nov 02 17:41:01.180464 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.180427    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3484eb28-e4be-4867-a6e6-aac7908e8124-maas-api-openshift-ca" pod "3484eb28-e4be-4867-a6e6-aac7908e8124" (UID: "3484eb28-e4be-4867-a6e6-aac7908e8124"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:41:01.180652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.180628    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3484eb28-e4be-4867-a6e6-aac7908e8124-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:01.182172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.182149    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3484eb28-e4be-4867-a6e6-aac7908e8124-maas-api-xks-ca" pod "3484eb28-e4be-4867-a6e6-aac7908e8124" (UID: "3484eb28-e4be-4867-a6e6-aac7908e8124"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:41:01.182288 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.182237    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3484eb28-e4be-4867-a6e6-aac7908e8124-kube-api-access-vqg4d" pod "3484eb28-e4be-4867-a6e6-aac7908e8124" (UID: "3484eb28-e4be-4867-a6e6-aac7908e8124"). InnerVolumeSpecName "kube-api-access-vqg4d". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:01.281883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.281848    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3484eb28-e4be-4867-a6e6-aac7908e8124-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:01.281883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.281886    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vqg4d\" (UniqueName: \"kubernetes.io/projected/3484eb28-e4be-4867-a6e6-aac7908e8124-kube-api-access-vqg4d\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:01.578414 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.578383    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"
Nov 02 17:41:01.578589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.578418    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj" event={"ID":"3484eb28-e4be-4867-a6e6-aac7908e8124","Type":"ContainerDied","Data":"415b02f3ea0b0e388232d59dc113c7e0cdc15d611a95f8b25db7cbface3e709b"}
Nov 02 17:41:01.578589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.578507    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="415b02f3ea0b0e388232d59dc113c7e0cdc15d611a95f8b25db7cbface3e709b"
Nov 02 17:41:01.580220 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.580184    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4" event={"ID":"a0f4e882-e05a-4ad5-8cfe-20117be35a46","Type":"ContainerStarted","Data":"2629f005a6d1a27d2d8f3d1800b2335b2c0713ddcadea2a0bc1196e58a3a324a"}
Nov 02 17:41:01.580357 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.580236    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4" event={"ID":"a0f4e882-e05a-4ad5-8cfe-20117be35a46","Type":"ContainerStarted","Data":"fdf2978bafea5d87f88e8bb944cabd1f472a85a75a8e6b5cd115c444ce591cb5"}
Nov 02 17:41:01.625567 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:01.625521    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4" podStartSLOduration=1.625508036 podStartE2EDuration="1.625508036s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:01.624281618 +0000 UTC m=+678.115344779" watchObservedRunningTime="2026-11-02 17:41:01.625508036 +0000 UTC m=+678.116571224"
Nov 02 17:41:02.588689 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:02.588651    4456 generic.go:356] "Generic (PLEG): container finished" podID="a0f4e882-e05a-4ad5-8cfe-20117be35a46" containerID="2629f005a6d1a27d2d8f3d1800b2335b2c0713ddcadea2a0bc1196e58a3a324a" exitCode=0
Nov 02 17:41:02.589128 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:02.588741    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4" event={"ID":"a0f4e882-e05a-4ad5-8cfe-20117be35a46","Type":"ContainerDied","Data":"2629f005a6d1a27d2d8f3d1800b2335b2c0713ddcadea2a0bc1196e58a3a324a"}
Nov 02 17:41:02.736979 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:02.736943    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:03.773362 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:03.773331    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"
Nov 02 17:41:03.811054 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:03.811016    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a0f4e882-e05a-4ad5-8cfe-20117be35a46-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f4e882-e05a-4ad5-8cfe-20117be35a46-maas-api-xks-ca\") pod \"a0f4e882-e05a-4ad5-8cfe-20117be35a46\" (UID: \"a0f4e882-e05a-4ad5-8cfe-20117be35a46\") "
Nov 02 17:41:03.811228 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:03.811156    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a0f4e882-e05a-4ad5-8cfe-20117be35a46-kube-api-access-fjft8\" (UniqueName: \"kubernetes.io/projected/a0f4e882-e05a-4ad5-8cfe-20117be35a46-kube-api-access-fjft8\") pod \"a0f4e882-e05a-4ad5-8cfe-20117be35a46\" (UID: \"a0f4e882-e05a-4ad5-8cfe-20117be35a46\") "
Nov 02 17:41:03.811228 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:03.811193    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a0f4e882-e05a-4ad5-8cfe-20117be35a46-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f4e882-e05a-4ad5-8cfe-20117be35a46-maas-api-openshift-ca\") pod \"a0f4e882-e05a-4ad5-8cfe-20117be35a46\" (UID: \"a0f4e882-e05a-4ad5-8cfe-20117be35a46\") "
Nov 02 17:41:03.811580 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:03.811553    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a0f4e882-e05a-4ad5-8cfe-20117be35a46-maas-api-openshift-ca" pod "a0f4e882-e05a-4ad5-8cfe-20117be35a46" (UID: "a0f4e882-e05a-4ad5-8cfe-20117be35a46"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:41:03.813213 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:03.813191    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a0f4e882-e05a-4ad5-8cfe-20117be35a46-maas-api-xks-ca" pod "a0f4e882-e05a-4ad5-8cfe-20117be35a46" (UID: "a0f4e882-e05a-4ad5-8cfe-20117be35a46"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:41:03.813375 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:03.813360    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a0f4e882-e05a-4ad5-8cfe-20117be35a46-kube-api-access-fjft8" pod "a0f4e882-e05a-4ad5-8cfe-20117be35a46" (UID: "a0f4e882-e05a-4ad5-8cfe-20117be35a46"). InnerVolumeSpecName "kube-api-access-fjft8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:03.912614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:03.912534    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fjft8\" (UniqueName: \"kubernetes.io/projected/a0f4e882-e05a-4ad5-8cfe-20117be35a46-kube-api-access-fjft8\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:03.912614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:03.912560    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a0f4e882-e05a-4ad5-8cfe-20117be35a46-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:03.912614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:03.912570    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a0f4e882-e05a-4ad5-8cfe-20117be35a46-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:04.601755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:04.601721    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"
Nov 02 17:41:04.601755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:04.601738    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4" event={"ID":"a0f4e882-e05a-4ad5-8cfe-20117be35a46","Type":"ContainerDied","Data":"fdf2978bafea5d87f88e8bb944cabd1f472a85a75a8e6b5cd115c444ce591cb5"}
Nov 02 17:41:04.601979 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:04.601827    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="fdf2978bafea5d87f88e8bb944cabd1f472a85a75a8e6b5cd115c444ce591cb5"
Nov 02 17:41:04.736922 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:04.736883    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:06.736106 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:06.736000    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:07.054194 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.054116    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"]
Nov 02 17:41:07.054659 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.054644    4456 cpu_manager.go:414] "containerMap: removing container" podUID="3484eb28-e4be-4867-a6e6-aac7908e8124" containerName="revoke-keys"
Nov 02 17:41:07.054704 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.054661    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3484eb28-e4be-4867-a6e6-aac7908e8124" containerName="revoke-keys"
Nov 02 17:41:07.054704 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.054675    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a0f4e882-e05a-4ad5-8cfe-20117be35a46" containerName="revoke-keys"
Nov 02 17:41:07.054704 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.054680    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a0f4e882-e05a-4ad5-8cfe-20117be35a46" containerName="revoke-keys"
Nov 02 17:41:07.054799 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.054781    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="3484eb28-e4be-4867-a6e6-aac7908e8124" containerName="revoke-keys"
Nov 02 17:41:07.054799 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.054791    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a0f4e882-e05a-4ad5-8cfe-20117be35a46" containerName="revoke-keys"
Nov 02 17:41:07.058472 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.058446    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"
Nov 02 17:41:07.078969 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.077103    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"]
Nov 02 17:41:07.085364 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.085339    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:41:07.085464 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.085393    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:41:07.151725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.151693    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26efd847-04dd-4445-9f92-da25a98bd953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2250f85a-s246f\" (UID: \"26efd847-04dd-4445-9f92-da25a98bd953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"
Nov 02 17:41:07.151725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.151733    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d2tjp\" (UniqueName: \"kubernetes.io/projected/26efd847-04dd-4445-9f92-da25a98bd953-kube-api-access-d2tjp\") pod \"maas-api-revoke-sub-2250f85a-s246f\" (UID: \"26efd847-04dd-4445-9f92-da25a98bd953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"
Nov 02 17:41:07.151969 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.151948    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26efd847-04dd-4445-9f92-da25a98bd953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2250f85a-s246f\" (UID: \"26efd847-04dd-4445-9f92-da25a98bd953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"
Nov 02 17:41:07.252707 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.252673    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26efd847-04dd-4445-9f92-da25a98bd953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2250f85a-s246f\" (UID: \"26efd847-04dd-4445-9f92-da25a98bd953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"
Nov 02 17:41:07.252886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.252711    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d2tjp\" (UniqueName: \"kubernetes.io/projected/26efd847-04dd-4445-9f92-da25a98bd953-kube-api-access-d2tjp\") pod \"maas-api-revoke-sub-2250f85a-s246f\" (UID: \"26efd847-04dd-4445-9f92-da25a98bd953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"
Nov 02 17:41:07.252886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.252780    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26efd847-04dd-4445-9f92-da25a98bd953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2250f85a-s246f\" (UID: \"26efd847-04dd-4445-9f92-da25a98bd953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"
Nov 02 17:41:07.253515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.253480    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26efd847-04dd-4445-9f92-da25a98bd953-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2250f85a-s246f\" (UID: \"26efd847-04dd-4445-9f92-da25a98bd953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"
Nov 02 17:41:07.255311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.255293    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26efd847-04dd-4445-9f92-da25a98bd953-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2250f85a-s246f\" (UID: \"26efd847-04dd-4445-9f92-da25a98bd953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"
Nov 02 17:41:07.276655 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.276619    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d2tjp\" (UniqueName: \"kubernetes.io/projected/26efd847-04dd-4445-9f92-da25a98bd953-kube-api-access-d2tjp\") pod \"maas-api-revoke-sub-2250f85a-s246f\" (UID: \"26efd847-04dd-4445-9f92-da25a98bd953\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"
Nov 02 17:41:07.370720 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.370638    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"
Nov 02 17:41:07.675568 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:07.675545    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"]
Nov 02 17:41:07.677364 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:41:07.677336    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod26efd847_04dd_4445_9f92_da25a98bd953.slice/crio-0c202e818d2bc476b3ca840912c7f1bf7f7006334d4cc499d107b61feba8a7a2 WatchSource:0}: Error finding container 0c202e818d2bc476b3ca840912c7f1bf7f7006334d4cc499d107b61feba8a7a2: Status 404 returned error can't find the container with id 0c202e818d2bc476b3ca840912c7f1bf7f7006334d4cc499d107b61feba8a7a2
Nov 02 17:41:08.626748 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:08.626709    4456 generic.go:356] "Generic (PLEG): container finished" podID="26efd847-04dd-4445-9f92-da25a98bd953" containerID="c345096ba2dafb1e748d9c9b439faaf1c551d47f8e33d5de9d48d1e39cc3c87b" exitCode=0
Nov 02 17:41:08.627166 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:08.626803    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f" event={"ID":"26efd847-04dd-4445-9f92-da25a98bd953","Type":"ContainerDied","Data":"c345096ba2dafb1e748d9c9b439faaf1c551d47f8e33d5de9d48d1e39cc3c87b"}
Nov 02 17:41:08.627166 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:08.626896    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f" event={"ID":"26efd847-04dd-4445-9f92-da25a98bd953","Type":"ContainerStarted","Data":"0c202e818d2bc476b3ca840912c7f1bf7f7006334d4cc499d107b61feba8a7a2"}
Nov 02 17:41:08.752012 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:08.751697    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:09.801131 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:09.801092    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"
Nov 02 17:41:09.885945 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:09.885692    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/26efd847-04dd-4445-9f92-da25a98bd953-kube-api-access-d2tjp\" (UniqueName: \"kubernetes.io/projected/26efd847-04dd-4445-9f92-da25a98bd953-kube-api-access-d2tjp\") pod \"26efd847-04dd-4445-9f92-da25a98bd953\" (UID: \"26efd847-04dd-4445-9f92-da25a98bd953\") "
Nov 02 17:41:09.885945 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:09.885784    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/26efd847-04dd-4445-9f92-da25a98bd953-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26efd847-04dd-4445-9f92-da25a98bd953-maas-api-xks-ca\") pod \"26efd847-04dd-4445-9f92-da25a98bd953\" (UID: \"26efd847-04dd-4445-9f92-da25a98bd953\") "
Nov 02 17:41:09.885945 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:09.885870    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/26efd847-04dd-4445-9f92-da25a98bd953-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26efd847-04dd-4445-9f92-da25a98bd953-maas-api-openshift-ca\") pod \"26efd847-04dd-4445-9f92-da25a98bd953\" (UID: \"26efd847-04dd-4445-9f92-da25a98bd953\") "
Nov 02 17:41:09.886867 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:09.886797    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/26efd847-04dd-4445-9f92-da25a98bd953-maas-api-openshift-ca" pod "26efd847-04dd-4445-9f92-da25a98bd953" (UID: "26efd847-04dd-4445-9f92-da25a98bd953"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:41:09.889565 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:09.889537    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/26efd847-04dd-4445-9f92-da25a98bd953-maas-api-xks-ca" pod "26efd847-04dd-4445-9f92-da25a98bd953" (UID: "26efd847-04dd-4445-9f92-da25a98bd953"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:41:09.890133 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:09.890107    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/26efd847-04dd-4445-9f92-da25a98bd953-kube-api-access-d2tjp" pod "26efd847-04dd-4445-9f92-da25a98bd953" (UID: "26efd847-04dd-4445-9f92-da25a98bd953"). InnerVolumeSpecName "kube-api-access-d2tjp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:09.987549 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:09.987510    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d2tjp\" (UniqueName: \"kubernetes.io/projected/26efd847-04dd-4445-9f92-da25a98bd953-kube-api-access-d2tjp\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:09.987549 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:09.987539    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/26efd847-04dd-4445-9f92-da25a98bd953-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:09.987549 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:09.987548    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/26efd847-04dd-4445-9f92-da25a98bd953-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:10.642513 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:10.642483    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"
Nov 02 17:41:10.642712 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:10.642477    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f" event={"ID":"26efd847-04dd-4445-9f92-da25a98bd953","Type":"ContainerDied","Data":"0c202e818d2bc476b3ca840912c7f1bf7f7006334d4cc499d107b61feba8a7a2"}
Nov 02 17:41:10.642712 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:10.642617    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0c202e818d2bc476b3ca840912c7f1bf7f7006334d4cc499d107b61feba8a7a2"
Nov 02 17:41:10.740347 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:10.740314    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:12.077775 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.077734    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"]
Nov 02 17:41:12.079794 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.079761    4456 cpu_manager.go:414] "containerMap: removing container" podUID="26efd847-04dd-4445-9f92-da25a98bd953" containerName="revoke-keys"
Nov 02 17:41:12.079794 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.079794    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="26efd847-04dd-4445-9f92-da25a98bd953" containerName="revoke-keys"
Nov 02 17:41:12.080128 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.080102    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="26efd847-04dd-4445-9f92-da25a98bd953" containerName="revoke-keys"
Nov 02 17:41:12.083998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.083968    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"
Nov 02 17:41:12.101605 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.101574    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"]
Nov 02 17:41:12.109405 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.109381    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:41:12.109546 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.109454    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:41:12.111142 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.111119    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaba299d-6qjfx\" (UID: \"1ab4274b-2b63-4b09-88e3-62e77c2c40a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"
Nov 02 17:41:12.111261 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.111231    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaba299d-6qjfx\" (UID: \"1ab4274b-2b63-4b09-88e3-62e77c2c40a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"
Nov 02 17:41:12.111261 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.111260    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nss9t\" (UniqueName: \"kubernetes.io/projected/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-kube-api-access-nss9t\") pod \"maas-api-revoke-sub-aaba299d-6qjfx\" (UID: \"1ab4274b-2b63-4b09-88e3-62e77c2c40a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"
Nov 02 17:41:12.212427 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.212388    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nss9t\" (UniqueName: \"kubernetes.io/projected/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-kube-api-access-nss9t\") pod \"maas-api-revoke-sub-aaba299d-6qjfx\" (UID: \"1ab4274b-2b63-4b09-88e3-62e77c2c40a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"
Nov 02 17:41:12.212608 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.212520    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaba299d-6qjfx\" (UID: \"1ab4274b-2b63-4b09-88e3-62e77c2c40a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"
Nov 02 17:41:12.212698 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.212674    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaba299d-6qjfx\" (UID: \"1ab4274b-2b63-4b09-88e3-62e77c2c40a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"
Nov 02 17:41:12.213477 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.213449    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-aaba299d-6qjfx\" (UID: \"1ab4274b-2b63-4b09-88e3-62e77c2c40a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"
Nov 02 17:41:12.215125 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.215104    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-maas-api-xks-ca\") pod \"maas-api-revoke-sub-aaba299d-6qjfx\" (UID: \"1ab4274b-2b63-4b09-88e3-62e77c2c40a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"
Nov 02 17:41:12.236338 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.236314    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nss9t\" (UniqueName: \"kubernetes.io/projected/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-kube-api-access-nss9t\") pod \"maas-api-revoke-sub-aaba299d-6qjfx\" (UID: \"1ab4274b-2b63-4b09-88e3-62e77c2c40a6\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"
Nov 02 17:41:12.403427 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.403351    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"
Nov 02 17:41:12.656101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.656020    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:12.714278 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:12.714248    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"]
Nov 02 17:41:12.716879 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:41:12.716854    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1ab4274b_2b63_4b09_88e3_62e77c2c40a6.slice/crio-7c10bc5264bba54c0582d8e47a5e313db7ec754b742d145eca41484afa583d63 WatchSource:0}: Error finding container 7c10bc5264bba54c0582d8e47a5e313db7ec754b742d145eca41484afa583d63: Status 404 returned error can't find the container with id 7c10bc5264bba54c0582d8e47a5e313db7ec754b742d145eca41484afa583d63
Nov 02 17:41:13.663011 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:13.662958    4456 generic.go:356] "Generic (PLEG): container finished" podID="1ab4274b-2b63-4b09-88e3-62e77c2c40a6" containerID="cce8ed0bab1398a692794e320e465b8efe5133255dfe647a72d88c2494357fce" exitCode=0
Nov 02 17:41:13.663471 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:13.663054    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx" event={"ID":"1ab4274b-2b63-4b09-88e3-62e77c2c40a6","Type":"ContainerDied","Data":"cce8ed0bab1398a692794e320e465b8efe5133255dfe647a72d88c2494357fce"}
Nov 02 17:41:13.663471 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:13.663123    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx" event={"ID":"1ab4274b-2b63-4b09-88e3-62e77c2c40a6","Type":"ContainerStarted","Data":"7c10bc5264bba54c0582d8e47a5e313db7ec754b742d145eca41484afa583d63"}
Nov 02 17:41:13.782282 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:13.782248    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:14.576714 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:14.576653    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:15.076842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:15.075109    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"
Nov 02 17:41:15.234901 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:15.204186    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-maas-api-xks-ca\") pod \"1ab4274b-2b63-4b09-88e3-62e77c2c40a6\" (UID: \"1ab4274b-2b63-4b09-88e3-62e77c2c40a6\") "
Nov 02 17:41:15.234901 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:15.204263    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-kube-api-access-nss9t\" (UniqueName: \"kubernetes.io/projected/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-kube-api-access-nss9t\") pod \"1ab4274b-2b63-4b09-88e3-62e77c2c40a6\" (UID: \"1ab4274b-2b63-4b09-88e3-62e77c2c40a6\") "
Nov 02 17:41:15.234901 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:15.204359    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-maas-api-openshift-ca\") pod \"1ab4274b-2b63-4b09-88e3-62e77c2c40a6\" (UID: \"1ab4274b-2b63-4b09-88e3-62e77c2c40a6\") "
Nov 02 17:41:15.235551 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:15.235505    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-maas-api-openshift-ca" pod "1ab4274b-2b63-4b09-88e3-62e77c2c40a6" (UID: "1ab4274b-2b63-4b09-88e3-62e77c2c40a6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:41:15.269885 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:15.268911    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-maas-api-xks-ca" pod "1ab4274b-2b63-4b09-88e3-62e77c2c40a6" (UID: "1ab4274b-2b63-4b09-88e3-62e77c2c40a6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:41:15.319846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:15.311121    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:15.319846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:15.311155    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:15.319846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:15.313021    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-kube-api-access-nss9t" pod "1ab4274b-2b63-4b09-88e3-62e77c2c40a6" (UID: "1ab4274b-2b63-4b09-88e3-62e77c2c40a6"). InnerVolumeSpecName "kube-api-access-nss9t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:15.413200 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:15.413134    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nss9t\" (UniqueName: \"kubernetes.io/projected/1ab4274b-2b63-4b09-88e3-62e77c2c40a6-kube-api-access-nss9t\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:15.770840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:15.768162    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"
Nov 02 17:41:15.770840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:15.768241    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx" event={"ID":"1ab4274b-2b63-4b09-88e3-62e77c2c40a6","Type":"ContainerDied","Data":"7c10bc5264bba54c0582d8e47a5e313db7ec754b742d145eca41484afa583d63"}
Nov 02 17:41:15.770840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:15.768332    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7c10bc5264bba54c0582d8e47a5e313db7ec754b742d145eca41484afa583d63"
Nov 02 17:41:16.165279 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:41:16.165242    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:41:16.355544 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:16.355507    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:16.508443 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:16.508408    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:16.655475 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:16.655443    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:18.370603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:18.370565    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:20.102138 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.102097    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"]
Nov 02 17:41:20.102892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.102872    4456 cpu_manager.go:414] "containerMap: removing container" podUID="1ab4274b-2b63-4b09-88e3-62e77c2c40a6" containerName="revoke-keys"
Nov 02 17:41:20.102951 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.102895    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1ab4274b-2b63-4b09-88e3-62e77c2c40a6" containerName="revoke-keys"
Nov 02 17:41:20.103053 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.103042    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="1ab4274b-2b63-4b09-88e3-62e77c2c40a6" containerName="revoke-keys"
Nov 02 17:41:20.106335 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.106315    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"
Nov 02 17:41:20.135738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.135707    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:41:20.136585 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.136562    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:41:20.138916 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.138895    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"]
Nov 02 17:41:20.170233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.170192    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:20.192805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.192769    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739ac036-8e8a-4585-b03e-febe64f2f54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e564800-rzxqj\" (UID: \"739ac036-8e8a-4585-b03e-febe64f2f54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"
Nov 02 17:41:20.193041 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.192894    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739ac036-8e8a-4585-b03e-febe64f2f54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e564800-rzxqj\" (UID: \"739ac036-8e8a-4585-b03e-febe64f2f54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"
Nov 02 17:41:20.193041 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.192926    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k58nf\" (UniqueName: \"kubernetes.io/projected/739ac036-8e8a-4585-b03e-febe64f2f54b-kube-api-access-k58nf\") pod \"maas-api-revoke-sub-4e564800-rzxqj\" (UID: \"739ac036-8e8a-4585-b03e-febe64f2f54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"
Nov 02 17:41:20.294277 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.294235    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739ac036-8e8a-4585-b03e-febe64f2f54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e564800-rzxqj\" (UID: \"739ac036-8e8a-4585-b03e-febe64f2f54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"
Nov 02 17:41:20.294463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.294331    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739ac036-8e8a-4585-b03e-febe64f2f54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e564800-rzxqj\" (UID: \"739ac036-8e8a-4585-b03e-febe64f2f54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"
Nov 02 17:41:20.294463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.294371    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k58nf\" (UniqueName: \"kubernetes.io/projected/739ac036-8e8a-4585-b03e-febe64f2f54b-kube-api-access-k58nf\") pod \"maas-api-revoke-sub-4e564800-rzxqj\" (UID: \"739ac036-8e8a-4585-b03e-febe64f2f54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"
Nov 02 17:41:20.295053 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.294995    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739ac036-8e8a-4585-b03e-febe64f2f54b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4e564800-rzxqj\" (UID: \"739ac036-8e8a-4585-b03e-febe64f2f54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"
Nov 02 17:41:20.296809 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.296790    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739ac036-8e8a-4585-b03e-febe64f2f54b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4e564800-rzxqj\" (UID: \"739ac036-8e8a-4585-b03e-febe64f2f54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"
Nov 02 17:41:20.318402 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.318373    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k58nf\" (UniqueName: \"kubernetes.io/projected/739ac036-8e8a-4585-b03e-febe64f2f54b-kube-api-access-k58nf\") pod \"maas-api-revoke-sub-4e564800-rzxqj\" (UID: \"739ac036-8e8a-4585-b03e-febe64f2f54b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"
Nov 02 17:41:20.419264 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.419182    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"
Nov 02 17:41:20.732607 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.732575    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"]
Nov 02 17:41:20.799404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:20.799372    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj" event={"ID":"739ac036-8e8a-4585-b03e-febe64f2f54b","Type":"ContainerStarted","Data":"52783183a35e5d29b9ad87e027f33996950d4b2e8f84bec03cdbe9afbf873ab8"}
Nov 02 17:41:21.807752 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:21.807661    4456 generic.go:356] "Generic (PLEG): container finished" podID="739ac036-8e8a-4585-b03e-febe64f2f54b" containerID="58815a74d6cd4ebb7a553793ea8bb01fee47fc2d24f1ddd4222cf9349e10f1fd" exitCode=0
Nov 02 17:41:21.808135 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:21.807765    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj" event={"ID":"739ac036-8e8a-4585-b03e-febe64f2f54b","Type":"ContainerDied","Data":"58815a74d6cd4ebb7a553793ea8bb01fee47fc2d24f1ddd4222cf9349e10f1fd"}
Nov 02 17:41:21.875085 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:21.875045    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:22.965376 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:22.965351    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"
Nov 02 17:41:23.030033 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.029993    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/739ac036-8e8a-4585-b03e-febe64f2f54b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739ac036-8e8a-4585-b03e-febe64f2f54b-maas-api-xks-ca\") pod \"739ac036-8e8a-4585-b03e-febe64f2f54b\" (UID: \"739ac036-8e8a-4585-b03e-febe64f2f54b\") "
Nov 02 17:41:23.030241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.030093    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/739ac036-8e8a-4585-b03e-febe64f2f54b-kube-api-access-k58nf\" (UniqueName: \"kubernetes.io/projected/739ac036-8e8a-4585-b03e-febe64f2f54b-kube-api-access-k58nf\") pod \"739ac036-8e8a-4585-b03e-febe64f2f54b\" (UID: \"739ac036-8e8a-4585-b03e-febe64f2f54b\") "
Nov 02 17:41:23.030344 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.030268    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/739ac036-8e8a-4585-b03e-febe64f2f54b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739ac036-8e8a-4585-b03e-febe64f2f54b-maas-api-openshift-ca\") pod \"739ac036-8e8a-4585-b03e-febe64f2f54b\" (UID: \"739ac036-8e8a-4585-b03e-febe64f2f54b\") "
Nov 02 17:41:23.030956 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.030878    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/739ac036-8e8a-4585-b03e-febe64f2f54b-maas-api-openshift-ca" pod "739ac036-8e8a-4585-b03e-febe64f2f54b" (UID: "739ac036-8e8a-4585-b03e-febe64f2f54b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:41:23.032872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.032808    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/739ac036-8e8a-4585-b03e-febe64f2f54b-kube-api-access-k58nf" pod "739ac036-8e8a-4585-b03e-febe64f2f54b" (UID: "739ac036-8e8a-4585-b03e-febe64f2f54b"). InnerVolumeSpecName "kube-api-access-k58nf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:23.033284 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.033241    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/739ac036-8e8a-4585-b03e-febe64f2f54b-maas-api-xks-ca" pod "739ac036-8e8a-4585-b03e-febe64f2f54b" (UID: "739ac036-8e8a-4585-b03e-febe64f2f54b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:41:23.050366 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.050326    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:23.132310 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.132229    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/739ac036-8e8a-4585-b03e-febe64f2f54b-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:23.132310 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.132262    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k58nf\" (UniqueName: \"kubernetes.io/projected/739ac036-8e8a-4585-b03e-febe64f2f54b-kube-api-access-k58nf\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:23.132310 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.132276    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/739ac036-8e8a-4585-b03e-febe64f2f54b-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:23.202165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.202127    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:23.675719 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.675669    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:23.797201 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.797173    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"]
Nov 02 17:41:23.797712 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.797699    4456 cpu_manager.go:414] "containerMap: removing container" podUID="739ac036-8e8a-4585-b03e-febe64f2f54b" containerName="revoke-keys"
Nov 02 17:41:23.797763 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.797715    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="739ac036-8e8a-4585-b03e-febe64f2f54b" containerName="revoke-keys"
Nov 02 17:41:23.797887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.797873    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="739ac036-8e8a-4585-b03e-febe64f2f54b" containerName="revoke-keys"
Nov 02 17:41:23.800671 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.800654    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"
Nov 02 17:41:23.820044 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.820018    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"]
Nov 02 17:41:23.828853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.828809    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"
Nov 02 17:41:23.828992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.828863    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj" event={"ID":"739ac036-8e8a-4585-b03e-febe64f2f54b","Type":"ContainerDied","Data":"52783183a35e5d29b9ad87e027f33996950d4b2e8f84bec03cdbe9afbf873ab8"}
Nov 02 17:41:23.828992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.828911    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="52783183a35e5d29b9ad87e027f33996950d4b2e8f84bec03cdbe9afbf873ab8"
Nov 02 17:41:23.941919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.941887    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4705529b-3538-410f-9504-b2d8fe857615-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9453ae6a-sgnwb\" (UID: \"4705529b-3538-410f-9504-b2d8fe857615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"
Nov 02 17:41:23.941919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.941922    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rh9ws\" (UniqueName: \"kubernetes.io/projected/4705529b-3538-410f-9504-b2d8fe857615-kube-api-access-rh9ws\") pod \"maas-api-revoke-sub-9453ae6a-sgnwb\" (UID: \"4705529b-3538-410f-9504-b2d8fe857615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"
Nov 02 17:41:23.942132 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:23.942101    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4705529b-3538-410f-9504-b2d8fe857615-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9453ae6a-sgnwb\" (UID: \"4705529b-3538-410f-9504-b2d8fe857615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"
Nov 02 17:41:24.043601 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:24.043565    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4705529b-3538-410f-9504-b2d8fe857615-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9453ae6a-sgnwb\" (UID: \"4705529b-3538-410f-9504-b2d8fe857615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"
Nov 02 17:41:24.043601 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:24.043602    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rh9ws\" (UniqueName: \"kubernetes.io/projected/4705529b-3538-410f-9504-b2d8fe857615-kube-api-access-rh9ws\") pod \"maas-api-revoke-sub-9453ae6a-sgnwb\" (UID: \"4705529b-3538-410f-9504-b2d8fe857615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"
Nov 02 17:41:24.044064 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:24.043707    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4705529b-3538-410f-9504-b2d8fe857615-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9453ae6a-sgnwb\" (UID: \"4705529b-3538-410f-9504-b2d8fe857615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"
Nov 02 17:41:24.053252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:24.044754    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4705529b-3538-410f-9504-b2d8fe857615-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-9453ae6a-sgnwb\" (UID: \"4705529b-3538-410f-9504-b2d8fe857615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"
Nov 02 17:41:24.053252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:24.047283    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4705529b-3538-410f-9504-b2d8fe857615-maas-api-xks-ca\") pod \"maas-api-revoke-sub-9453ae6a-sgnwb\" (UID: \"4705529b-3538-410f-9504-b2d8fe857615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"
Nov 02 17:41:24.069116 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:24.069090    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rh9ws\" (UniqueName: \"kubernetes.io/projected/4705529b-3538-410f-9504-b2d8fe857615-kube-api-access-rh9ws\") pod \"maas-api-revoke-sub-9453ae6a-sgnwb\" (UID: \"4705529b-3538-410f-9504-b2d8fe857615\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"
Nov 02 17:41:24.112763 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:24.112730    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"
Nov 02 17:41:24.417317 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:24.417283    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"]
Nov 02 17:41:24.418312 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:41:24.418285    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4705529b_3538_410f_9504_b2d8fe857615.slice/crio-bf3f1036e694a6c0c9d6c0bfe5a587f4c7468d45da29b9c6dfdb0e0aa52cfa76 WatchSource:0}: Error finding container bf3f1036e694a6c0c9d6c0bfe5a587f4c7468d45da29b9c6dfdb0e0aa52cfa76: Status 404 returned error can't find the container with id bf3f1036e694a6c0c9d6c0bfe5a587f4c7468d45da29b9c6dfdb0e0aa52cfa76
Nov 02 17:41:24.836800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:24.836761    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb" event={"ID":"4705529b-3538-410f-9504-b2d8fe857615","Type":"ContainerStarted","Data":"ec233d01041de7bed6a6112b2c49a5fc01c5e6f2af99a81f226bb11368e94607"}
Nov 02 17:41:24.836800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:24.836831    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb" event={"ID":"4705529b-3538-410f-9504-b2d8fe857615","Type":"ContainerStarted","Data":"bf3f1036e694a6c0c9d6c0bfe5a587f4c7468d45da29b9c6dfdb0e0aa52cfa76"}
Nov 02 17:41:24.894221 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:24.894170    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb" podStartSLOduration=1.894155474 podStartE2EDuration="1.894155474s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:24.893250055 +0000 UTC m=+701.384313236" watchObservedRunningTime="2026-11-02 17:41:24.894155474 +0000 UTC m=+701.385218634"
Nov 02 17:41:25.551884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:25.551787    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:25.844316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:25.844232    4456 generic.go:356] "Generic (PLEG): container finished" podID="4705529b-3538-410f-9504-b2d8fe857615" containerID="ec233d01041de7bed6a6112b2c49a5fc01c5e6f2af99a81f226bb11368e94607" exitCode=0
Nov 02 17:41:25.844316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:25.844263    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb" event={"ID":"4705529b-3538-410f-9504-b2d8fe857615","Type":"ContainerDied","Data":"ec233d01041de7bed6a6112b2c49a5fc01c5e6f2af99a81f226bb11368e94607"}
Nov 02 17:41:27.017078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:27.017049    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"
Nov 02 17:41:27.081100 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:27.081066    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/4705529b-3538-410f-9504-b2d8fe857615-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4705529b-3538-410f-9504-b2d8fe857615-maas-api-xks-ca\") pod \"4705529b-3538-410f-9504-b2d8fe857615\" (UID: \"4705529b-3538-410f-9504-b2d8fe857615\") "
Nov 02 17:41:27.081289 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:27.081175    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4705529b-3538-410f-9504-b2d8fe857615-kube-api-access-rh9ws\" (UniqueName: \"kubernetes.io/projected/4705529b-3538-410f-9504-b2d8fe857615-kube-api-access-rh9ws\") pod \"4705529b-3538-410f-9504-b2d8fe857615\" (UID: \"4705529b-3538-410f-9504-b2d8fe857615\") "
Nov 02 17:41:27.081289 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:27.081220    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4705529b-3538-410f-9504-b2d8fe857615-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4705529b-3538-410f-9504-b2d8fe857615-maas-api-openshift-ca\") pod \"4705529b-3538-410f-9504-b2d8fe857615\" (UID: \"4705529b-3538-410f-9504-b2d8fe857615\") "
Nov 02 17:41:27.081646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:27.081616    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4705529b-3538-410f-9504-b2d8fe857615-maas-api-openshift-ca" pod "4705529b-3538-410f-9504-b2d8fe857615" (UID: "4705529b-3538-410f-9504-b2d8fe857615"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:41:27.081852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:27.081735    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/4705529b-3538-410f-9504-b2d8fe857615-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:27.083463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:27.083435    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/4705529b-3538-410f-9504-b2d8fe857615-maas-api-xks-ca" pod "4705529b-3538-410f-9504-b2d8fe857615" (UID: "4705529b-3538-410f-9504-b2d8fe857615"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:41:27.084371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:27.084345    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4705529b-3538-410f-9504-b2d8fe857615-kube-api-access-rh9ws" pod "4705529b-3538-410f-9504-b2d8fe857615" (UID: "4705529b-3538-410f-9504-b2d8fe857615"). InnerVolumeSpecName "kube-api-access-rh9ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:41:27.183104 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:27.183082    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/4705529b-3538-410f-9504-b2d8fe857615-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:27.183246 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:27.183108    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rh9ws\" (UniqueName: \"kubernetes.io/projected/4705529b-3538-410f-9504-b2d8fe857615-kube-api-access-rh9ws\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:41:27.370680 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:27.370301    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:27.859000 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:27.858951    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb" event={"ID":"4705529b-3538-410f-9504-b2d8fe857615","Type":"ContainerDied","Data":"bf3f1036e694a6c0c9d6c0bfe5a587f4c7468d45da29b9c6dfdb0e0aa52cfa76"}
Nov 02 17:41:27.859197 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:27.859053    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bf3f1036e694a6c0c9d6c0bfe5a587f4c7468d45da29b9c6dfdb0e0aa52cfa76"
Nov 02 17:41:27.859197 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:27.859138    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"
Nov 02 17:41:29.167097 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:29.167063    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:31.086372 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:31.086334    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:32.892404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:32.892370    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:33.309565 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:33.309499    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:33.466061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:33.466025    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:34.745781 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:34.745740    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:36.675989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:36.675954    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:38.357235 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:38.357200    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:40.164527 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:40.164489    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:41.970110 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:41.970059    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:43.755578 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:43.755532    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:45.554598 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:45.554561    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:46.170949 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:41:46.170874    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:41:46.350463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.350423    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"]
Nov 02 17:41:46.351271 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.351216    4456 cpu_manager.go:414] "containerMap: removing container" podUID="4705529b-3538-410f-9504-b2d8fe857615" containerName="revoke-keys"
Nov 02 17:41:46.351271 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.351238    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4705529b-3538-410f-9504-b2d8fe857615" containerName="revoke-keys"
Nov 02 17:41:46.351705 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.351385    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="4705529b-3538-410f-9504-b2d8fe857615" containerName="revoke-keys"
Nov 02 17:41:46.354661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.354640    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.361515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.361487    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"]
Nov 02 17:41:46.364896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.364873    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.386206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.386176    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sd4jt\" (UniqueName: \"kubernetes.io/projected/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-kube-api-access-sd4jt\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.386423 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.386401    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.386589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.386572    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.386724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.386708    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.386862 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.386844    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.387038 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.387020    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.387151 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.387137    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.387256 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.387241    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.387366 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.387351    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.388018 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.387999    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-798kx\" (UniqueName: \"kubernetes.io/projected/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-kube-api-access-798kx\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.388181 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.388163    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.388301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.388286    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.388412 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.388397    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.388530 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.388516    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.388681 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.388665    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.389330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.388805    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.389494 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.389475    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.389616 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.389599    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.390014 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.388999    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"]
Nov 02 17:41:46.395590 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.395565    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"]
Nov 02 17:41:46.491427 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.491378    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sd4jt\" (UniqueName: \"kubernetes.io/projected/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-kube-api-access-sd4jt\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.491620 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.491449    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.491620 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.491492    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.491620 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.491529    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.491620 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.491552    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.491620 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.491618    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.491935 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.491643    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.491935 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.491671    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.491935 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.491708    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.491935 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.491734    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-798kx\" (UniqueName: \"kubernetes.io/projected/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-kube-api-access-798kx\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.491935 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.491776    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.491935 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.491831    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.491935 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.491862    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.491935 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.491914    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.492359 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.491996    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.492359 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.492053    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.492359 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.492092    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.492359 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.492117    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.492359 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.492286    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.492619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.492409    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.492619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.492526    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.492857 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.492798    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.493733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.492970    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.493872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.493230    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.493872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.493316    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.493872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.493557    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.493872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.493675    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.493872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.493840    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.504489 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.504439    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.504589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.504552    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.505169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.505144    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.505793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.505767    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.516598 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.516530    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.518333 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.518304    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.519750 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.519713    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-798kx\" (UniqueName: \"kubernetes.io/projected/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-kube-api-access-798kx\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:46.519908 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.519888    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sd4jt\" (UniqueName: \"kubernetes.io/projected/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-kube-api-access-sd4jt\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.674944 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.674913    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:46.703011 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:46.702982    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:47.018164 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:47.018140    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"]
Nov 02 17:41:47.018625 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:41:47.018606    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod426b74e4_82bb_46fc_b57f_f6ae93bcbefc.slice/crio-5865296a299bec7320dfdf5c8a7916d8c4a2a286029ce23c3b738fed503f57bd WatchSource:0}: Error finding container 5865296a299bec7320dfdf5c8a7916d8c4a2a286029ce23c3b738fed503f57bd: Status 404 returned error can't find the container with id 5865296a299bec7320dfdf5c8a7916d8c4a2a286029ce23c3b738fed503f57bd
Nov 02 17:41:47.021032 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:47.021004    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:47.021101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:47.021066    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:47.021101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:47.021092    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:47.044847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:47.044808    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"]
Nov 02 17:41:47.053975 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:41:47.053953    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a700a1f_2010_4ca5_a78b_4cae0c8b4d42.slice/crio-5e1574f81545ba1e233d0f6e36aad2b74f57ac8fb1c4e690469415bdfe862e21 WatchSource:0}: Error finding container 5e1574f81545ba1e233d0f6e36aad2b74f57ac8fb1c4e690469415bdfe862e21: Status 404 returned error can't find the container with id 5e1574f81545ba1e233d0f6e36aad2b74f57ac8fb1c4e690469415bdfe862e21
Nov 02 17:41:47.056319 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:47.056290    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:47.056384 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:47.056354    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:47.056418 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:47.056384    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:41:47.136731 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:47.136692    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g" event={"ID":"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42","Type":"ContainerStarted","Data":"5e1574f81545ba1e233d0f6e36aad2b74f57ac8fb1c4e690469415bdfe862e21"}
Nov 02 17:41:47.138745 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:47.138713    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh" event={"ID":"426b74e4-82bb-46fc-b57f-f6ae93bcbefc","Type":"ContainerStarted","Data":"ac1d6511fe0a00fd60087a38695324eeed479fc719e25ab20d0cbc5b050d78e3"}
Nov 02 17:41:47.138884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:47.138764    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh" event={"ID":"426b74e4-82bb-46fc-b57f-f6ae93bcbefc","Type":"ContainerStarted","Data":"5865296a299bec7320dfdf5c8a7916d8c4a2a286029ce23c3b738fed503f57bd"}
Nov 02 17:41:47.195140 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:47.195082    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh" podStartSLOduration=1.195061925 podStartE2EDuration="1.195061925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:47.193185981 +0000 UTC m=+723.684249224" watchObservedRunningTime="2026-11-02 17:41:47.195061925 +0000 UTC m=+723.686125086"
Nov 02 17:41:47.376889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:47.376525    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:47.675589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:47.675497    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:48.151793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:48.151758    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g" event={"ID":"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42","Type":"ContainerStarted","Data":"a62cadeb98b862ad8413a6f6bc577441b9d503e25033aa1deeac9e06e9d18660"}
Nov 02 17:41:48.240483 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:48.240405    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g" podStartSLOduration=2.240387859 podStartE2EDuration="2.240387859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:41:48.217460351 +0000 UTC m=+724.708523517" watchObservedRunningTime="2026-11-02 17:41:48.240387859 +0000 UTC m=+724.731451071"
Nov 02 17:41:48.677957 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:48.677913    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 17:41:48.678531 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:48.678505    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh" podUID="426b74e4-82bb-46fc-b57f-f6ae93bcbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 17:41:48.712077 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:48.712044    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:48.714004 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:48.713929    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 17:41:48.714215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:48.714187    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g" podUID="8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 17:41:49.243191 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:49.243160    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:49.675284 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:49.675197    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 17:41:49.675284 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:49.675252    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh" podUID="426b74e4-82bb-46fc-b57f-f6ae93bcbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 17:41:49.707504 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:49.707210    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 17:41:49.707504 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:49.707282    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g" podUID="8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 17:41:50.243890 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:50.242392    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:50.676356 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:50.675988    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 17:41:50.676356 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:50.676038    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh" podUID="426b74e4-82bb-46fc-b57f-f6ae93bcbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 17:41:50.705253 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:50.705213    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 17:41:50.705525 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:50.705498    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g" podUID="8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 17:41:51.676728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:51.676694    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 17:41:51.677293 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:51.677267    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh" podUID="426b74e4-82bb-46fc-b57f-f6ae93bcbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 17:41:51.703693 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:51.703661    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 17:41:51.703978 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:51.703950    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g" podUID="8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 17:41:52.540586 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:52.540551    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:52.675191 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:52.675150    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 17:41:52.675389 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:52.675208    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh" podUID="426b74e4-82bb-46fc-b57f-f6ae93bcbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 17:41:52.703573 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:52.703540    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused" start-of-body=
Nov 02 17:41:52.704233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:52.704188    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g" podUID="8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.65:15021/healthz/ready\": dial tcp 10.134.0.65:15021: connect: connection refused"
Nov 02 17:41:53.675298 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:53.675246    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused" start-of-body=
Nov 02 17:41:53.675507 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:53.675320    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh" podUID="426b74e4-82bb-46fc-b57f-f6ae93bcbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": dial tcp 10.134.0.64:15021: connect: connection refused"
Nov 02 17:41:54.053995 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:54.053856    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:54.054589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:54.054571    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:54.056367 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:54.056320    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:41:54.620227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:54.620198    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:54.680475 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:54.680445    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:54.681362 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:54.681339    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:54.682119 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:54.682099    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:41:54.847469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:54.847435    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:56.369172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:56.369134    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:56.793576 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:56.793542    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:58.205333 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:58.205265    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:59.255885 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.255843    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"]
Nov 02 17:41:59.258688 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.258672    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.288839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.287757    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"distinct-w3-5eedab-kserve-self-signed-certs\""
Nov 02 17:41:59.293483 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.293452    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"]
Nov 02 17:41:59.362295 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.362254    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:41:59.373586 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.373558    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-tmp-dir\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.373721 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.373595    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0874d0e8-5883-477d-b64a-d455934cfc90-tls-certs\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.373721 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.373629    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-home\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.373798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.373745    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-model-cache\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.373852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.373834    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vfwdc\" (UniqueName: \"kubernetes.io/projected/0874d0e8-5883-477d-b64a-d455934cfc90-kube-api-access-vfwdc\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.373975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.373962    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-dshm\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.477392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.477344    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-model-cache\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.477597 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.477458    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vfwdc\" (UniqueName: \"kubernetes.io/projected/0874d0e8-5883-477d-b64a-d455934cfc90-kube-api-access-vfwdc\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.477665 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.477612    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-dshm\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.477722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.477699    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-tmp-dir\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.477779 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.477734    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0874d0e8-5883-477d-b64a-d455934cfc90-tls-certs\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.477951 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.477773    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-home\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.478384 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.478355    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-home\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.478799 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.478640    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-model-cache\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.481076 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.481049    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-tmp-dir\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.489880 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.489807    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-dshm\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.497464 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.497436    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0874d0e8-5883-477d-b64a-d455934cfc90-tls-certs\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.503195 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.503171    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vfwdc\" (UniqueName: \"kubernetes.io/projected/0874d0e8-5883-477d-b64a-d455934cfc90-kube-api-access-vfwdc\") pod \"distinct-w3-5eedab-kserve-856b8c7458-k2lgz\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") " pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.576931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.576850    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:41:59.884590 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:41:59.884557    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"]
Nov 02 17:41:59.886051 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:41:59.886020    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod0874d0e8_5883_477d_b64a_d455934cfc90.slice/crio-a3900d4e7b8bd64d510d5e5a0f6580e0160fb9f87f5221cc11fdbf0df3354938 WatchSource:0}: Error finding container a3900d4e7b8bd64d510d5e5a0f6580e0160fb9f87f5221cc11fdbf0df3354938: Status 404 returned error can't find the container with id a3900d4e7b8bd64d510d5e5a0f6580e0160fb9f87f5221cc11fdbf0df3354938
Nov 02 17:42:00.151130 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:00.151047    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:00.277421 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:00.277385    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" event={"ID":"0874d0e8-5883-477d-b64a-d455934cfc90","Type":"ContainerStarted","Data":"d2c0ed473e942d6f113bde4b458bb80e06031b1d79f122c6822f5c1c0be919b4"}
Nov 02 17:42:00.277421 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:00.277435    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" event={"ID":"0874d0e8-5883-477d-b64a-d455934cfc90","Type":"ContainerStarted","Data":"a3900d4e7b8bd64d510d5e5a0f6580e0160fb9f87f5221cc11fdbf0df3354938"}
Nov 02 17:42:00.322219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:00.322168    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podStartSLOduration=1.322154019 podStartE2EDuration="1.322154019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:41:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:00.319858579 +0000 UTC m=+736.810921743" watchObservedRunningTime="2026-11-02 17:42:00.322154019 +0000 UTC m=+736.813217187"
Nov 02 17:42:00.577908 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:00.577875    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:42:00.962103 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:00.962073    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"]
Nov 02 17:42:00.964956 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:00.964931    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:00.993547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:00.993522    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"]
Nov 02 17:42:01.026484 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.026449    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.026649 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.026501    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.026649 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.026589    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.026649 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.026620    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3891e06c-0c53-421c-b379-c38efca65905-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.026828 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.026679    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3891e06c-0c53-421c-b379-c38efca65905-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.026828 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.026720    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.026828 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.026787    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.027001 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.026836    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3891e06c-0c53-421c-b379-c38efca65905-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.027001 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.026871    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-btsdk\" (UniqueName: \"kubernetes.io/projected/3891e06c-0c53-421c-b379-c38efca65905-kube-api-access-btsdk\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.127641 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.127600    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.127870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.127654    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3891e06c-0c53-421c-b379-c38efca65905-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.127870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.127713    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3891e06c-0c53-421c-b379-c38efca65905-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.127870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.127756    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.127870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.127840    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.128115 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.127871    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3891e06c-0c53-421c-b379-c38efca65905-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.128115 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.127909    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-btsdk\" (UniqueName: \"kubernetes.io/projected/3891e06c-0c53-421c-b379-c38efca65905-kube-api-access-btsdk\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.128115 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.127978    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.128115 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.128013    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.128313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.128169    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.128664 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.128639    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.128795 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.128722    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.129206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.129183    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3891e06c-0c53-421c-b379-c38efca65905-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.129313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.129263    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.130945 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.130922    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.131169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.131151    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3891e06c-0c53-421c-b379-c38efca65905-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.154692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.154664    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3891e06c-0c53-421c-b379-c38efca65905-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.154981 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.154753    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-btsdk\" (UniqueName: \"kubernetes.io/projected/3891e06c-0c53-421c-b379-c38efca65905-kube-api-access-btsdk\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-6ct96\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.278599 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.278509    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:01.313534 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.313503    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"]
Nov 02 17:42:01.317291 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.317267    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.335176 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.332358    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.335176 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.332512    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.335176 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.332562    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d9a093-015c-498b-8808-7939c01499da-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.335176 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.332607    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7zvs6\" (UniqueName: \"kubernetes.io/projected/c1d9a093-015c-498b-8808-7939c01499da-kube-api-access-7zvs6\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.335176 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.332666    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.335176 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.332736    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d9a093-015c-498b-8808-7939c01499da-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.335176 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.332779    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.335176 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.332841    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d9a093-015c-498b-8808-7939c01499da-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.335176 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.332875    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.342947 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.342916    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"]
Nov 02 17:42:01.434326 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.434274    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d9a093-015c-498b-8808-7939c01499da-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.434515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.434339    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.434515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.434427    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d9a093-015c-498b-8808-7939c01499da-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.434515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.434460    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.434701 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.434559    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.434757 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.434701    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.434757 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.434745    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d9a093-015c-498b-8808-7939c01499da-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.434914 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.434765    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.434914 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.434787    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7zvs6\" (UniqueName: \"kubernetes.io/projected/c1d9a093-015c-498b-8808-7939c01499da-kube-api-access-7zvs6\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.434914 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.434853    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.435082 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.434919    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.435179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.435155    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.435458 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.435430    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d9a093-015c-498b-8808-7939c01499da-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.435713 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.435688    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.437953 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.437923    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.438141 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.438124    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d9a093-015c-498b-8808-7939c01499da-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.458558 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.458534    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d9a093-015c-498b-8808-7939c01499da-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.458709 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.458689    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7zvs6\" (UniqueName: \"kubernetes.io/projected/c1d9a093-015c-498b-8808-7939c01499da-kube-api-access-7zvs6\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.630236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.630210    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"]
Nov 02 17:42:01.631560 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:42:01.631532    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3891e06c_0c53_421c_b379_c38efca65905.slice/crio-2318d551484065b666e190c0c034c38972b8723fb968f8039c1421518bc39ab1 WatchSource:0}: Error finding container 2318d551484065b666e190c0c034c38972b8723fb968f8039c1421518bc39ab1: Status 404 returned error can't find the container with id 2318d551484065b666e190c0c034c38972b8723fb968f8039c1421518bc39ab1
Nov 02 17:42:01.634097 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.634066    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:01.634201 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.634134    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:01.634201 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.634182    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:01.654654 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.654631    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:01.969163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.969124    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:02.007201 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:01.998705    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"]
Nov 02 17:42:02.020587 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:42:02.020285    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc1d9a093_015c_498b_8808_7939c01499da.slice/crio-fd00914ee78a8ce938a74edc9a6ab41beddba2384c41c6b7134e9b0e33273c0c WatchSource:0}: Error finding container fd00914ee78a8ce938a74edc9a6ab41beddba2384c41c6b7134e9b0e33273c0c: Status 404 returned error can't find the container with id fd00914ee78a8ce938a74edc9a6ab41beddba2384c41c6b7134e9b0e33273c0c
Nov 02 17:42:02.024151 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:02.024104    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:02.024374 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:02.024349    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:02.024499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:02.024479    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:02.295469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:02.295362    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq" event={"ID":"c1d9a093-015c-498b-8808-7939c01499da","Type":"ContainerStarted","Data":"dac0a2352ac1baae2810fea9acc9b8c90d9dee012f5df6f86ddb0a56f3c33341"}
Nov 02 17:42:02.295469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:02.295429    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq" event={"ID":"c1d9a093-015c-498b-8808-7939c01499da","Type":"ContainerStarted","Data":"fd00914ee78a8ce938a74edc9a6ab41beddba2384c41c6b7134e9b0e33273c0c"}
Nov 02 17:42:02.298187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:02.298158    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96" event={"ID":"3891e06c-0c53-421c-b379-c38efca65905","Type":"ContainerStarted","Data":"fb40301125fc84beda005c90f6eecf66143cedc0c31f6edeedcdf53df1b482b3"}
Nov 02 17:42:02.298356 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:02.298208    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96" event={"ID":"3891e06c-0c53-421c-b379-c38efca65905","Type":"ContainerStarted","Data":"2318d551484065b666e190c0c034c38972b8723fb968f8039c1421518bc39ab1"}
Nov 02 17:42:02.345841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:02.345738    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq" podStartSLOduration=1.3457215439999999 podStartE2EDuration="1.345721544s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:02.341426277 +0000 UTC m=+738.832489437" watchObservedRunningTime="2026-11-02 17:42:02.345721544 +0000 UTC m=+738.836784705"
Nov 02 17:42:02.388150 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:02.388093    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96" podStartSLOduration=2.388078683 podStartE2EDuration="2.388078683s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:02.387716373 +0000 UTC m=+738.878779552" watchObservedRunningTime="2026-11-02 17:42:02.388078683 +0000 UTC m=+738.879141846"
Nov 02 17:42:02.655540 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:02.655442    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:02.769216 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:02.769184    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:03.279554 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:03.279527    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:03.297843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:03.296018    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 17:42:03.297843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:03.296113    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96" podUID="3891e06c-0c53-421c-b379-c38efca65905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 17:42:03.665173 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:03.664912    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 17:42:03.665173 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:03.665016    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq" podUID="c1d9a093-015c-498b-8808-7939c01499da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 17:42:03.895750 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:03.895714    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:04.283843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:04.279965    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 17:42:04.283843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:04.280045    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96" podUID="3891e06c-0c53-421c-b379-c38efca65905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 17:42:04.655856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:04.655742    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 17:42:04.656235 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:04.655863    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq" podUID="c1d9a093-015c-498b-8808-7939c01499da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 17:42:05.279521 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:05.279485    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 17:42:05.279674 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:05.279550    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96" podUID="3891e06c-0c53-421c-b379-c38efca65905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 17:42:05.655862 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:05.655767    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 17:42:05.656032 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:05.655870    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq" podUID="c1d9a093-015c-498b-8808-7939c01499da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 17:42:05.804583 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:05.804514    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:06.279354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:06.279317    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 17:42:06.279506 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:06.279365    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96" podUID="3891e06c-0c53-421c-b379-c38efca65905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 17:42:06.655229 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:06.655131    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 17:42:06.655229 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:06.655192    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq" podUID="c1d9a093-015c-498b-8808-7939c01499da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 17:42:07.279302 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:07.279268    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 17:42:07.279800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:07.279330    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96" podUID="3891e06c-0c53-421c-b379-c38efca65905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 17:42:07.575471 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:07.575363    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:07.656312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:07.656270    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 17:42:07.656615 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:07.656587    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq" podUID="c1d9a093-015c-498b-8808-7939c01499da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 17:42:08.279210 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:08.279171    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused" start-of-body=
Nov 02 17:42:08.279422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:08.279246    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96" podUID="3891e06c-0c53-421c-b379-c38efca65905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": dial tcp 10.134.0.67:15021: connect: connection refused"
Nov 02 17:42:08.655652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:08.655550    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused" start-of-body=
Nov 02 17:42:08.655652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:08.655627    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq" podUID="c1d9a093-015c-498b-8808-7939c01499da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": dial tcp 10.134.0.68:15021: connect: connection refused"
Nov 02 17:42:09.284203 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:09.284170    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:09.284685 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:09.284628    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:09.285475 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:09.285452    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:42:09.345649 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:09.345617    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:09.577845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:09.577738    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:42:09.591238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:09.591207    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:42:09.659629 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:09.659599    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:09.660070 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:09.660051    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:09.660731 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:09.660715    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:42:10.404487 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:10.404455    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:42:11.189802 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:11.189644    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:12.941573 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:12.941533    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:14.741247 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:14.741217    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:15.899989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:15.899915    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"]
Nov 02 17:42:15.971934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:15.971902    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:15.972989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:15.972963    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"]
Nov 02 17:42:16.012934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.012884    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:16.077204 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.076462    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.077204 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.076602    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.077204 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.076633    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.077204 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.076665    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4323971-4e6e-43f6-bc29-49cbe5f12949-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.077204 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.076737    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.077204 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.076780    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-26s6h\" (UniqueName: \"kubernetes.io/projected/a4323971-4e6e-43f6-bc29-49cbe5f12949-kube-api-access-26s6h\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.077204 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.076838    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.077204 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.076892    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.077204 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.076939    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.178740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.178640    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.178740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.178690    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.178740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.178727    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4323971-4e6e-43f6-bc29-49cbe5f12949-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.179082 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.178794    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.179082 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.178849    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-26s6h\" (UniqueName: \"kubernetes.io/projected/a4323971-4e6e-43f6-bc29-49cbe5f12949-kube-api-access-26s6h\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.179082 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.178893    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.179082 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.178944    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.179082 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.178986    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.179082 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.179080    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.184832 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.179938    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.184832 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.182722    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.184832 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.182740    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.184832 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.182998    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.184832 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.183233    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.184832 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.184081    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4323971-4e6e-43f6-bc29-49cbe5f12949-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.184832 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:42:16.184758    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:42:16.185379 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.185337    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.207031 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.207000    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.272851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.252569    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-26s6h\" (UniqueName: \"kubernetes.io/projected/a4323971-4e6e-43f6-bc29-49cbe5f12949-kube-api-access-26s6h\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.310248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.301117    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:16.407709 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.407654    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"]
Nov 02 17:42:16.418099 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.418064    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"]
Nov 02 17:42:16.418250 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.418179    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.534509 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.534468    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.545837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.544663    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.545837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.544851    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.545837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.544889    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.545837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.544952    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.545837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.544993    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-56xc8\" (UniqueName: \"kubernetes.io/projected/35675ac4-845c-4700-95c7-9f3ba72bd20a-kube-api-access-56xc8\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.545837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.545055    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.545837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.545088    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35675ac4-845c-4700-95c7-9f3ba72bd20a-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.545837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.545162    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.680765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.649377    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.680765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.649516    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.680765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.649549    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.680765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.649608    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.680765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.649644    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-56xc8\" (UniqueName: \"kubernetes.io/projected/35675ac4-845c-4700-95c7-9f3ba72bd20a-kube-api-access-56xc8\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.680765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.649698    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.680765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.649730    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35675ac4-845c-4700-95c7-9f3ba72bd20a-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.680765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.649793    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.680765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.649890    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.680765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.650233    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.680765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.659114    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.680765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.659714    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.680765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.660344    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35675ac4-845c-4700-95c7-9f3ba72bd20a-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.680765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.660572    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.722840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.722155    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:16.963843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:16.956782    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:17.085024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:17.082334    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:17.085024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:17.082444    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:17.085024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:17.082490    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:17.170842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:17.165128    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"]
Nov 02 17:42:17.176841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:17.172418    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:17.237260 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:17.237173    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-56xc8\" (UniqueName: \"kubernetes.io/projected/35675ac4-845c-4700-95c7-9f3ba72bd20a-kube-api-access-56xc8\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:17.261933 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:17.243316    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:17.386366 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:17.386075    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:17.526540 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:17.526442    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" event={"ID":"a4323971-4e6e-43f6-bc29-49cbe5f12949","Type":"ContainerStarted","Data":"8b5a1359dbfc4bb66e3c6efb38ae746ffd5da85d3ce23a5e316fa1b4ccb878ce"}
Nov 02 17:42:17.687233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:17.686175    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:17.831075 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:17.831017    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:17.995839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:17.984556    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:18.091889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:18.085957    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:18.459844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:18.456918    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:18.459844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:18.457344    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 17:42:18.562847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:18.561202    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 02 17:42:18.769896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:18.748980    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:18.769896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:18.757563    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:42:18.791898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:18.779616    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:42:18.853184 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:18.838279    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:42:19.103284 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:18.913581    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"]
Nov 02 17:42:19.103284 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:19.091172    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" event={"ID":"a4323971-4e6e-43f6-bc29-49cbe5f12949","Type":"ContainerStarted","Data":"6398cfda523a8a8e2e673b041a862c3046065aa1b5573d8b8c734cf872ee2401"}
Nov 02 17:42:19.103284 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:19.091278    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:42:19.103284 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:19.091347    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:42:19.103284 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:18.997266    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:19.103284 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:19.097699    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:19.103284 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:19.097871    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:19.103284 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:18.996162    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:19.104169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:19.103408    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" podStartSLOduration=4.103388063 podStartE2EDuration="4.103388063s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:19.089772218 +0000 UTC m=+755.580835396" watchObservedRunningTime="2026-11-02 17:42:19.103388063 +0000 UTC m=+755.594451223"
Nov 02 17:42:19.317920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:19.311304    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:19.321768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:19.321727    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 17:42:19.322076 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:19.322023    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 17:42:19.425076 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:19.425049    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:42:19.927847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:19.927788    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs" event={"ID":"35675ac4-845c-4700-95c7-9f3ba72bd20a","Type":"ContainerStarted","Data":"45d088cb196f174e304b13054a3ac9be6e5ffbe0a3e532e8ace5a77828a452ff"}
Nov 02 17:42:19.928036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:19.927872    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs" event={"ID":"35675ac4-845c-4700-95c7-9f3ba72bd20a","Type":"ContainerStarted","Data":"6852a35a5d2c5faa28d093dff795ed3b7e2296bbdd025379045f5e200449179c"}
Nov 02 17:42:19.994840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:19.989421    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs" podStartSLOduration=3.98939585 podStartE2EDuration="3.98939585s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:19.976119464 +0000 UTC m=+756.467182662" watchObservedRunningTime="2026-11-02 17:42:19.98939585 +0000 UTC m=+756.480459020"
Nov 02 17:42:20.302246 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:20.302205    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 17:42:20.302848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:20.302792    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 17:42:20.391840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:20.389202    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:20.413841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:20.412966    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 17:42:20.413841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:20.413041    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs" podUID="35675ac4-845c-4700-95c7-9f3ba72bd20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 17:42:20.988747 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:20.988710    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:21.305809 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:21.305705    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 17:42:21.306467 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:21.306411    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 17:42:21.389442 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:21.389402    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 17:42:21.389754 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:21.389698    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs" podUID="35675ac4-845c-4700-95c7-9f3ba72bd20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 17:42:22.303240 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:22.303193    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 17:42:22.303488 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:22.303263    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 17:42:22.392255 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:22.392217    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 17:42:22.392863 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:22.392802    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs" podUID="35675ac4-845c-4700-95c7-9f3ba72bd20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 17:42:22.872691 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:22.872662    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:23.302716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:23.302242    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 17:42:23.302716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:23.302317    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 17:42:23.389392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:23.389353    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 17:42:23.389692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:23.389648    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs" podUID="35675ac4-845c-4700-95c7-9f3ba72bd20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 17:42:24.302843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:24.302278    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 17:42:24.302843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:24.302345    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 17:42:24.391843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:24.391400    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 17:42:24.391843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:24.391480    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs" podUID="35675ac4-845c-4700-95c7-9f3ba72bd20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 17:42:24.664205 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:24.664114    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:25.302843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:25.302209    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused" start-of-body=
Nov 02 17:42:25.302843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:25.302282    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": dial tcp 10.134.0.69:15021: connect: connection refused"
Nov 02 17:42:25.387898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:25.387237    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 17:42:25.387898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:25.387301    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs" podUID="35675ac4-845c-4700-95c7-9f3ba72bd20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 17:42:26.308369 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:26.308321    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:26.309079 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:26.309053    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:26.310074 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:26.310056    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 17:42:26.387266 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:26.387213    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused" start-of-body=
Nov 02 17:42:26.387432 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:26.387333    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs" podUID="35675ac4-845c-4700-95c7-9f3ba72bd20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": dial tcp 10.134.0.70:15021: connect: connection refused"
Nov 02 17:42:26.451545 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:26.451458    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:27.392583 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:27.392553    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:27.393066 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:27.393040    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:27.393752 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:27.393728    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:42:28.230534 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:28.230493    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:28.381548 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:28.381512    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:30.056256 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:30.056226    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:31.764252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:31.764201    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:33.590446 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:33.590402    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:33.789470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:33.789391    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:34.290843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:34.290776    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"]
Nov 02 17:42:34.305370 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:34.305342    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"
Nov 02 17:42:34.324804 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:34.324757    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:42:34.335844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:34.333912    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"]
Nov 02 17:42:34.346383 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:34.346336    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:42:34.470015 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:34.469978    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53281d37-399f-4063-a91e-b802ca02e1ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0af0d7a-bks96\" (UID: \"53281d37-399f-4063-a91e-b802ca02e1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"
Nov 02 17:42:34.470163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:34.470116    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53281d37-399f-4063-a91e-b802ca02e1ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0af0d7a-bks96\" (UID: \"53281d37-399f-4063-a91e-b802ca02e1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"
Nov 02 17:42:34.470224 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:34.470163    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ksvtw\" (UniqueName: \"kubernetes.io/projected/53281d37-399f-4063-a91e-b802ca02e1ff-kube-api-access-ksvtw\") pod \"maas-api-revoke-sub-f0af0d7a-bks96\" (UID: \"53281d37-399f-4063-a91e-b802ca02e1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"
Nov 02 17:42:34.571832 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:34.571722    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53281d37-399f-4063-a91e-b802ca02e1ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0af0d7a-bks96\" (UID: \"53281d37-399f-4063-a91e-b802ca02e1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"
Nov 02 17:42:34.572012 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:34.571856    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53281d37-399f-4063-a91e-b802ca02e1ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0af0d7a-bks96\" (UID: \"53281d37-399f-4063-a91e-b802ca02e1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"
Nov 02 17:42:34.572012 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:34.571914    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ksvtw\" (UniqueName: \"kubernetes.io/projected/53281d37-399f-4063-a91e-b802ca02e1ff-kube-api-access-ksvtw\") pod \"maas-api-revoke-sub-f0af0d7a-bks96\" (UID: \"53281d37-399f-4063-a91e-b802ca02e1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"
Nov 02 17:42:34.572557 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:34.572537    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53281d37-399f-4063-a91e-b802ca02e1ff-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f0af0d7a-bks96\" (UID: \"53281d37-399f-4063-a91e-b802ca02e1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"
Nov 02 17:42:34.574421 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:34.574397    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53281d37-399f-4063-a91e-b802ca02e1ff-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f0af0d7a-bks96\" (UID: \"53281d37-399f-4063-a91e-b802ca02e1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"
Nov 02 17:42:34.615717 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:34.615690    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ksvtw\" (UniqueName: \"kubernetes.io/projected/53281d37-399f-4063-a91e-b802ca02e1ff-kube-api-access-ksvtw\") pod \"maas-api-revoke-sub-f0af0d7a-bks96\" (UID: \"53281d37-399f-4063-a91e-b802ca02e1ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"
Nov 02 17:42:34.675676 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:34.675643    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"
Nov 02 17:42:35.032853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:35.032807    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"]
Nov 02 17:42:35.034614 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:42:35.034575    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod53281d37_399f_4063_a91e_b802ca02e1ff.slice/crio-496141030ad1104d2d40d7718eea5437b96dec1c14712c26e03dc997bc98f942 WatchSource:0}: Error finding container 496141030ad1104d2d40d7718eea5437b96dec1c14712c26e03dc997bc98f942: Status 404 returned error can't find the container with id 496141030ad1104d2d40d7718eea5437b96dec1c14712c26e03dc997bc98f942
Nov 02 17:42:35.094628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:35.094589    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96" event={"ID":"53281d37-399f-4063-a91e-b802ca02e1ff","Type":"ContainerStarted","Data":"496141030ad1104d2d40d7718eea5437b96dec1c14712c26e03dc997bc98f942"}
Nov 02 17:42:35.695078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:35.695028    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:36.105760 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:36.105668    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96" event={"ID":"53281d37-399f-4063-a91e-b802ca02e1ff","Type":"ContainerStarted","Data":"0dfa4b4d3d083585f98bf2bea0a07a716bca677f6de2e9b57f1682e3548cadde"}
Nov 02 17:42:36.149068 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:36.149017    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96" podStartSLOduration=2.149003881 podStartE2EDuration="2.149003881s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:36.148177383 +0000 UTC m=+772.639240567" watchObservedRunningTime="2026-11-02 17:42:36.149003881 +0000 UTC m=+772.640067082"
Nov 02 17:42:37.115073 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:37.114977    4456 generic.go:356] "Generic (PLEG): container finished" podID="53281d37-399f-4063-a91e-b802ca02e1ff" containerID="0dfa4b4d3d083585f98bf2bea0a07a716bca677f6de2e9b57f1682e3548cadde" exitCode=0
Nov 02 17:42:37.115508 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:37.115061    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96" event={"ID":"53281d37-399f-4063-a91e-b802ca02e1ff","Type":"ContainerDied","Data":"0dfa4b4d3d083585f98bf2bea0a07a716bca677f6de2e9b57f1682e3548cadde"}
Nov 02 17:42:37.197644 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:37.197608    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:37.485147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:37.485106    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:38.292373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:38.292340    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"
Nov 02 17:42:38.431627 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:38.431531    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/53281d37-399f-4063-a91e-b802ca02e1ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53281d37-399f-4063-a91e-b802ca02e1ff-maas-api-openshift-ca\") pod \"53281d37-399f-4063-a91e-b802ca02e1ff\" (UID: \"53281d37-399f-4063-a91e-b802ca02e1ff\") "
Nov 02 17:42:38.431627 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:38.431593    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/53281d37-399f-4063-a91e-b802ca02e1ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53281d37-399f-4063-a91e-b802ca02e1ff-maas-api-xks-ca\") pod \"53281d37-399f-4063-a91e-b802ca02e1ff\" (UID: \"53281d37-399f-4063-a91e-b802ca02e1ff\") "
Nov 02 17:42:38.431893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:38.431770    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/53281d37-399f-4063-a91e-b802ca02e1ff-kube-api-access-ksvtw\" (UniqueName: \"kubernetes.io/projected/53281d37-399f-4063-a91e-b802ca02e1ff-kube-api-access-ksvtw\") pod \"53281d37-399f-4063-a91e-b802ca02e1ff\" (UID: \"53281d37-399f-4063-a91e-b802ca02e1ff\") "
Nov 02 17:42:38.432023 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:38.431999    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/53281d37-399f-4063-a91e-b802ca02e1ff-maas-api-openshift-ca" pod "53281d37-399f-4063-a91e-b802ca02e1ff" (UID: "53281d37-399f-4063-a91e-b802ca02e1ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:42:38.432301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:38.432281    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/53281d37-399f-4063-a91e-b802ca02e1ff-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:42:38.434453 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:38.434432    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/53281d37-399f-4063-a91e-b802ca02e1ff-maas-api-xks-ca" pod "53281d37-399f-4063-a91e-b802ca02e1ff" (UID: "53281d37-399f-4063-a91e-b802ca02e1ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:42:38.434622 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:38.434605    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/53281d37-399f-4063-a91e-b802ca02e1ff-kube-api-access-ksvtw" pod "53281d37-399f-4063-a91e-b802ca02e1ff" (UID: "53281d37-399f-4063-a91e-b802ca02e1ff"). InnerVolumeSpecName "kube-api-access-ksvtw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:42:38.533169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:38.533130    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/53281d37-399f-4063-a91e-b802ca02e1ff-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:42:38.533169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:38.533160    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ksvtw\" (UniqueName: \"kubernetes.io/projected/53281d37-399f-4063-a91e-b802ca02e1ff-kube-api-access-ksvtw\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:42:39.131024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:39.130996    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"
Nov 02 17:42:39.131291 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:39.131020    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96" event={"ID":"53281d37-399f-4063-a91e-b802ca02e1ff","Type":"ContainerDied","Data":"496141030ad1104d2d40d7718eea5437b96dec1c14712c26e03dc997bc98f942"}
Nov 02 17:42:39.131291 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:39.131081    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="496141030ad1104d2d40d7718eea5437b96dec1c14712c26e03dc997bc98f942"
Nov 02 17:42:39.361801 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:39.361758    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:41.355660 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:41.355625    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:43.167000 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:43.166962    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:45.120071 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:45.120027    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:46.165669 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:42:46.165631    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:42:46.334828 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.334779    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"]
Nov 02 17:42:46.335508 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.335485    4456 cpu_manager.go:414] "containerMap: removing container" podUID="53281d37-399f-4063-a91e-b802ca02e1ff" containerName="revoke-keys"
Nov 02 17:42:46.335508 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.335504    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="53281d37-399f-4063-a91e-b802ca02e1ff" containerName="revoke-keys"
Nov 02 17:42:46.335730 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.335663    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="53281d37-399f-4063-a91e-b802ca02e1ff" containerName="revoke-keys"
Nov 02 17:42:46.342708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.341705    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.348270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.348240    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"]
Nov 02 17:42:46.357914 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.356232    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.380182 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.380154    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"]
Nov 02 17:42:46.391345 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.391320    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"]
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.416998    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"]
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.417121    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.434794    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.434901    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.434936    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5775b5a6-6407-46f4-84ef-96c708d7934b-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.435011    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.435039    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.435084    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.435107    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.435132    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67472e82-1ea4-420c-81cd-91952f3d0bf7-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.435195    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.435219    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.435253    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.435289    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.435326    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.435370    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.435435    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqv2l\" (UniqueName: \"kubernetes.io/projected/5775b5a6-6407-46f4-84ef-96c708d7934b-kube-api-access-nqv2l\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.435481    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.435506    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mnkj\" (UniqueName: \"kubernetes.io/projected/67472e82-1ea4-420c-81cd-91952f3d0bf7-kube-api-access-2mnkj\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.440923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.435537    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.504511 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.503973    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"]
Nov 02 17:42:46.537436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.537385    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.537619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.537440    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t42n8\" (UniqueName: \"kubernetes.io/projected/17f59d36-8a89-4385-bdaf-40777f5e1a4e-kube-api-access-t42n8\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.537619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.537486    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.537619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.537533    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.537619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.537558    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.537619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.537583    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67472e82-1ea4-420c-81cd-91952f3d0bf7-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.537619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.537607    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.537968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.537674    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.537968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.537699    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.537968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.537742    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.537968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.537781    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.537968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.537837    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.537968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.537867    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.537968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.537904    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.538337 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.538323    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.538591 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.538568    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.538677 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.538637    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67472e82-1ea4-420c-81cd-91952f3d0bf7-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.538762 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.538743    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.538903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.538884    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.539139 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.539122    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.539718 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.539692    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.539960 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.539938    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.540002 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.539965    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.540002 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.539984    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqv2l\" (UniqueName: \"kubernetes.io/projected/5775b5a6-6407-46f4-84ef-96c708d7934b-kube-api-access-nqv2l\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.540069 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.540026    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.540069 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.540063    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.540129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.540089    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mnkj\" (UniqueName: \"kubernetes.io/projected/67472e82-1ea4-420c-81cd-91952f3d0bf7-kube-api-access-2mnkj\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.540129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.540120    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.540199 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.540175    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.540233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.540219    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.540316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.540300    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.540358 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.540344    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5775b5a6-6407-46f4-84ef-96c708d7934b-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.540424 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.540410    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.540482 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.540468    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.541085 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.541060    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.541719 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.541530    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.542487 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.542466    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5775b5a6-6407-46f4-84ef-96c708d7934b-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.543240 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.543218    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.543342 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.543248    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.544428 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.544405    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.544515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.544405    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.563116 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.563093    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.563853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.563791    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.566041 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.566019    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqv2l\" (UniqueName: \"kubernetes.io/projected/5775b5a6-6407-46f4-84ef-96c708d7934b-kube-api-access-nqv2l\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.566531 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.566497    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mnkj\" (UniqueName: \"kubernetes.io/projected/67472e82-1ea4-420c-81cd-91952f3d0bf7-kube-api-access-2mnkj\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.642011 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.641979    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t42n8\" (UniqueName: \"kubernetes.io/projected/17f59d36-8a89-4385-bdaf-40777f5e1a4e-kube-api-access-t42n8\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.642192 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.642039    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.642192 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.642115    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.642192 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.642137    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.642192 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.642173    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.642405 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.642195    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.642405 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.642247    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.642405 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.642366    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.642557 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.642410    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.642557 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.642498    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.642660 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.642562    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.643000 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.642954    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.643150 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.643128    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.643333 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.643299    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.646093 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.646061    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.653629 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.653431    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.665945 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.665915    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:46.666694 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.666667    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.666891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.666872    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t42n8\" (UniqueName: \"kubernetes.io/projected/17f59d36-8a89-4385-bdaf-40777f5e1a4e-kube-api-access-t42n8\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.702170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.702078    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:46.835447 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.835166    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:46.969675 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:46.969568    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:47.013637 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.011059    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"]
Nov 02 17:42:47.031648 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.031603    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:47.031771 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.031684    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:47.031771 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.031715    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:47.068327 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.068302    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"]
Nov 02 17:42:47.069449 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:42:47.069425    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5775b5a6_6407_46f4_84ef_96c708d7934b.slice/crio-724b2f34a5f83d96e3ed0b3f0e724fa1b26929da5345368ee4fe9957a36e154f WatchSource:0}: Error finding container 724b2f34a5f83d96e3ed0b3f0e724fa1b26929da5345368ee4fe9957a36e154f: Status 404 returned error can't find the container with id 724b2f34a5f83d96e3ed0b3f0e724fa1b26929da5345368ee4fe9957a36e154f
Nov 02 17:42:47.072116 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.072078    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:47.072238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.072159    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:47.072238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.072200    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:47.168626 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.168581    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"]
Nov 02 17:42:47.176620 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:42:47.175603    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod17f59d36_8a89_4385_bdaf_40777f5e1a4e.slice/crio-d0537c97f868c7de7a7043431bfb11b2334963de56e3b55028a8fe9d138b59f5 WatchSource:0}: Error finding container d0537c97f868c7de7a7043431bfb11b2334963de56e3b55028a8fe9d138b59f5: Status 404 returned error can't find the container with id d0537c97f868c7de7a7043431bfb11b2334963de56e3b55028a8fe9d138b59f5
Nov 02 17:42:47.179033 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.178996    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:47.179122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.179079    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:47.179159 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.179119    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:42:47.216856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.216067    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9" event={"ID":"67472e82-1ea4-420c-81cd-91952f3d0bf7","Type":"ContainerStarted","Data":"20f6e7886d8d5f395dba728a4d5be467700d43e94ae5e95c113038277f47ab29"}
Nov 02 17:42:47.216856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.216130    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9" event={"ID":"67472e82-1ea4-420c-81cd-91952f3d0bf7","Type":"ContainerStarted","Data":"496be73610e3e6410b4d5484d32c14b99222791e86e8baed864b11bbc7a4c9c3"}
Nov 02 17:42:47.218869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.218621    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t" event={"ID":"17f59d36-8a89-4385-bdaf-40777f5e1a4e","Type":"ContainerStarted","Data":"d0537c97f868c7de7a7043431bfb11b2334963de56e3b55028a8fe9d138b59f5"}
Nov 02 17:42:47.220725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.220598    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" event={"ID":"5775b5a6-6407-46f4-84ef-96c708d7934b","Type":"ContainerStarted","Data":"99a6168054233269cb6e8344874ad06a145a87f2aae8004e7910f961ede56963"}
Nov 02 17:42:47.220725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.220653    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" event={"ID":"5775b5a6-6407-46f4-84ef-96c708d7934b","Type":"ContainerStarted","Data":"724b2f34a5f83d96e3ed0b3f0e724fa1b26929da5345368ee4fe9957a36e154f"}
Nov 02 17:42:47.270939 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.270893    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9" podStartSLOduration=1.2708783399999999 podStartE2EDuration="1.27087834s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:47.265942901 +0000 UTC m=+783.757006076" watchObservedRunningTime="2026-11-02 17:42:47.27087834 +0000 UTC m=+783.761941572"
Nov 02 17:42:47.325708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.325645    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" podStartSLOduration=1.325625369 podStartE2EDuration="1.325625369s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:47.322639629 +0000 UTC m=+783.813702792" watchObservedRunningTime="2026-11-02 17:42:47.325625369 +0000 UTC m=+783.816688531"
Nov 02 17:42:47.666141 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.666058    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:47.702266 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:47.702229    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:48.267900 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:48.267866    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t" event={"ID":"17f59d36-8a89-4385-bdaf-40777f5e1a4e","Type":"ContainerStarted","Data":"349f7a8f4f162e2da516d281b9f4438830f8645cf63394b83ab66e7b973dc19a"}
Nov 02 17:42:48.365841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:48.354518    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t" podStartSLOduration=2.354496823 podStartE2EDuration="2.354496823s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:42:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:42:48.352731171 +0000 UTC m=+784.843794357" watchObservedRunningTime="2026-11-02 17:42:48.354496823 +0000 UTC m=+784.845559986"
Nov 02 17:42:48.672884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:48.669750    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:42:48.673516 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:48.673178    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9" podUID="67472e82-1ea4-420c-81cd-91952f3d0bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 17:42:48.710841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:48.704078    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:42:48.710841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:48.704145    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 17:42:48.845836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:48.842727    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:48.845836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:48.842893    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 17:42:48.845836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:48.842957    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t" podUID="17f59d36-8a89-4385-bdaf-40777f5e1a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 17:42:49.462884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:49.460768    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:49.738901 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:49.727376    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:49.843836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:49.842397    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:42:49.843836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:49.842483    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9" podUID="67472e82-1ea4-420c-81cd-91952f3d0bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 17:42:49.843836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:49.843114    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:42:49.843836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:49.843160    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 17:42:49.868837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:49.848730    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 17:42:49.868837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:49.848777    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t" podUID="17f59d36-8a89-4385-bdaf-40777f5e1a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 17:42:50.744993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:50.744957    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:42:50.745591 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:50.745559    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 17:42:50.795859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:50.792162    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:42:50.795859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:50.792322    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9" podUID="67472e82-1ea4-420c-81cd-91952f3d0bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 17:42:50.879766 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:50.879727    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 17:42:50.880085 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:50.880054    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t" podUID="17f59d36-8a89-4385-bdaf-40777f5e1a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 17:42:51.485843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:51.484975    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:51.503252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:51.500293    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:51.709843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:51.694148    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:42:51.709843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:51.694219    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9" podUID="67472e82-1ea4-420c-81cd-91952f3d0bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 17:42:51.800902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:51.785776    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:42:51.800902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:51.785853    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 17:42:51.881852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:51.879860    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 17:42:51.881852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:51.879920    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t" podUID="17f59d36-8a89-4385-bdaf-40777f5e1a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 17:42:51.981855 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:51.978990    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 17:42:52.073897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:52.072979    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:52.209175 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:52.205310    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:52.413896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:52.409771    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:52.521859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:52.510668    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:42:52.521859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:52.511434    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:42:52.551844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:52.538784    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:42:52.712842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:52.711268    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:42:52.712842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:52.711340    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9" podUID="67472e82-1ea4-420c-81cd-91952f3d0bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 17:42:52.712842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:52.712411    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:42:52.712842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:52.712451    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 17:42:52.857842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:52.848654    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 17:42:52.879955 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:52.874642    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t" podUID="17f59d36-8a89-4385-bdaf-40777f5e1a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 17:42:53.411206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:53.411159    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:53.412193 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:53.412168    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:42:53.452965 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:53.452925    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:53.479568 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:53.479524    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:53.662904 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:53.632429    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:42:53.706374 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:53.706340    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:42:53.706638 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:53.706607    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9" podUID="67472e82-1ea4-420c-81cd-91952f3d0bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 17:42:53.708594 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:53.708570    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:42:53.708755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:53.708736    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 17:42:53.742982 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:53.742922    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:53.868018 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:53.867957    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 17:42:53.868794 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:53.868752    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t" podUID="17f59d36-8a89-4385-bdaf-40777f5e1a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 17:42:53.970222 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:53.969560    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:42:54.284106 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:54.284021    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:54.691633 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:54.689195    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused" start-of-body=
Nov 02 17:42:54.691633 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:54.689261    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9" podUID="67472e82-1ea4-420c-81cd-91952f3d0bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": dial tcp 10.134.0.72:15021: connect: connection refused"
Nov 02 17:42:54.712169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:54.711929    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused" start-of-body=
Nov 02 17:42:54.712169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:54.711987    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": dial tcp 10.134.0.73:15021: connect: connection refused"
Nov 02 17:42:54.902900 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:54.837416    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused" start-of-body=
Nov 02 17:42:54.902900 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:54.837488    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t" podUID="17f59d36-8a89-4385-bdaf-40777f5e1a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": dial tcp 10.134.0.74:15021: connect: connection refused"
Nov 02 17:42:56.461217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:56.459920    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:56.528115 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:56.525207    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:56.528115 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:56.526766    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:56.528115 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:56.528061    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 17:42:56.546771 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:56.544904    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:56.564140 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:56.564108    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:56.568657 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:56.568610    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:56.618866 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:56.616370    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:42:56.620961 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:56.620035    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:42:56.903347 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:56.900781    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:42:56.903347 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:56.900863    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t" podUID="17f59d36-8a89-4385-bdaf-40777f5e1a4e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.74:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:57.412847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:57.404116    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:42:57.412847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:57.404199    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:57.423836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:57.420043    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:57.423836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:57.421444    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:57.447172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:57.447146    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:42:57.463301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:57.461774    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:42:57.463301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:57.462184    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:42:57.491836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:57.491785    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:42:58.475737 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:42:58.475697    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:00.418512 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:00.418482    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:02.163305 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:02.163253    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:03.946330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:03.946291    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:05.644381 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.644348    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:05.755841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.753069    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"]
Nov 02 17:43:05.763346 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.763320    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"
Nov 02 17:43:05.774436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.773975    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"]
Nov 02 17:43:05.781309 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.781105    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:43:05.781309 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.781253    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:43:05.819106 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.819065    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:05.855578 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.855516    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647ee669-772d-4e08-9cf2-73a204ff09fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a86d703e-h8wd2\" (UID: \"647ee669-772d-4e08-9cf2-73a204ff09fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"
Nov 02 17:43:05.855954 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.855650    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mf5hr\" (UniqueName: \"kubernetes.io/projected/647ee669-772d-4e08-9cf2-73a204ff09fe-kube-api-access-mf5hr\") pod \"maas-api-revoke-sub-a86d703e-h8wd2\" (UID: \"647ee669-772d-4e08-9cf2-73a204ff09fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"
Nov 02 17:43:05.855954 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.855769    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647ee669-772d-4e08-9cf2-73a204ff09fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a86d703e-h8wd2\" (UID: \"647ee669-772d-4e08-9cf2-73a204ff09fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"
Nov 02 17:43:05.957683 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.957651    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647ee669-772d-4e08-9cf2-73a204ff09fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a86d703e-h8wd2\" (UID: \"647ee669-772d-4e08-9cf2-73a204ff09fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"
Nov 02 17:43:05.957902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.957789    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647ee669-772d-4e08-9cf2-73a204ff09fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a86d703e-h8wd2\" (UID: \"647ee669-772d-4e08-9cf2-73a204ff09fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"
Nov 02 17:43:05.957956 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.957933    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mf5hr\" (UniqueName: \"kubernetes.io/projected/647ee669-772d-4e08-9cf2-73a204ff09fe-kube-api-access-mf5hr\") pod \"maas-api-revoke-sub-a86d703e-h8wd2\" (UID: \"647ee669-772d-4e08-9cf2-73a204ff09fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"
Nov 02 17:43:05.958720 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.958695    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647ee669-772d-4e08-9cf2-73a204ff09fe-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a86d703e-h8wd2\" (UID: \"647ee669-772d-4e08-9cf2-73a204ff09fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"
Nov 02 17:43:05.960894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.960872    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647ee669-772d-4e08-9cf2-73a204ff09fe-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a86d703e-h8wd2\" (UID: \"647ee669-772d-4e08-9cf2-73a204ff09fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"
Nov 02 17:43:05.989433 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:05.989402    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mf5hr\" (UniqueName: \"kubernetes.io/projected/647ee669-772d-4e08-9cf2-73a204ff09fe-kube-api-access-mf5hr\") pod \"maas-api-revoke-sub-a86d703e-h8wd2\" (UID: \"647ee669-772d-4e08-9cf2-73a204ff09fe\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"
Nov 02 17:43:06.127487 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:06.127455    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"
Nov 02 17:43:06.434784 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:43:06.434755    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod647ee669_772d_4e08_9cf2_73a204ff09fe.slice/crio-ac0ecf1782556195cb8115ee390202e1d72fd2f40abf2bf62dec3575d025fab5 WatchSource:0}: Error finding container ac0ecf1782556195cb8115ee390202e1d72fd2f40abf2bf62dec3575d025fab5: Status 404 returned error can't find the container with id ac0ecf1782556195cb8115ee390202e1d72fd2f40abf2bf62dec3575d025fab5
Nov 02 17:43:06.435433 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:06.435408    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"]
Nov 02 17:43:06.639012 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:06.638976    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2" event={"ID":"647ee669-772d-4e08-9cf2-73a204ff09fe","Type":"ContainerStarted","Data":"ac0ecf1782556195cb8115ee390202e1d72fd2f40abf2bf62dec3575d025fab5"}
Nov 02 17:43:07.647063 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:07.646965    4456 generic.go:356] "Generic (PLEG): container finished" podID="647ee669-772d-4e08-9cf2-73a204ff09fe" containerID="eea951ab2e361567d81c71f7178d26251ee9b619b9390993938bf8c8e5960c8d" exitCode=0
Nov 02 17:43:07.647426 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:07.647057    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2" event={"ID":"647ee669-772d-4e08-9cf2-73a204ff09fe","Type":"ContainerDied","Data":"eea951ab2e361567d81c71f7178d26251ee9b619b9390993938bf8c8e5960c8d"}
Nov 02 17:43:07.660911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:07.660887    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:08.822706 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:08.822675    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"
Nov 02 17:43:08.899268 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:08.899217    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/647ee669-772d-4e08-9cf2-73a204ff09fe-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647ee669-772d-4e08-9cf2-73a204ff09fe-maas-api-xks-ca\") pod \"647ee669-772d-4e08-9cf2-73a204ff09fe\" (UID: \"647ee669-772d-4e08-9cf2-73a204ff09fe\") "
Nov 02 17:43:08.899464 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:08.899381    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/647ee669-772d-4e08-9cf2-73a204ff09fe-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647ee669-772d-4e08-9cf2-73a204ff09fe-maas-api-openshift-ca\") pod \"647ee669-772d-4e08-9cf2-73a204ff09fe\" (UID: \"647ee669-772d-4e08-9cf2-73a204ff09fe\") "
Nov 02 17:43:08.899464 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:08.899436    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/647ee669-772d-4e08-9cf2-73a204ff09fe-kube-api-access-mf5hr\" (UniqueName: \"kubernetes.io/projected/647ee669-772d-4e08-9cf2-73a204ff09fe-kube-api-access-mf5hr\") pod \"647ee669-772d-4e08-9cf2-73a204ff09fe\" (UID: \"647ee669-772d-4e08-9cf2-73a204ff09fe\") "
Nov 02 17:43:08.900060 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:08.900030    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/647ee669-772d-4e08-9cf2-73a204ff09fe-maas-api-openshift-ca" pod "647ee669-772d-4e08-9cf2-73a204ff09fe" (UID: "647ee669-772d-4e08-9cf2-73a204ff09fe"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:08.903007 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:08.902966    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/647ee669-772d-4e08-9cf2-73a204ff09fe-maas-api-xks-ca" pod "647ee669-772d-4e08-9cf2-73a204ff09fe" (UID: "647ee669-772d-4e08-9cf2-73a204ff09fe"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:43:08.903129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:08.903073    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/647ee669-772d-4e08-9cf2-73a204ff09fe-kube-api-access-mf5hr" pod "647ee669-772d-4e08-9cf2-73a204ff09fe" (UID: "647ee669-772d-4e08-9cf2-73a204ff09fe"). InnerVolumeSpecName "kube-api-access-mf5hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:08.940891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:08.940858    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"]
Nov 02 17:43:08.942003 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:08.941965    4456 cpu_manager.go:414] "containerMap: removing container" podUID="647ee669-772d-4e08-9cf2-73a204ff09fe" containerName="revoke-keys"
Nov 02 17:43:08.942167 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:08.942151    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="647ee669-772d-4e08-9cf2-73a204ff09fe" containerName="revoke-keys"
Nov 02 17:43:08.942475 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:08.942457    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="647ee669-772d-4e08-9cf2-73a204ff09fe" containerName="revoke-keys"
Nov 02 17:43:08.946761 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:08.946736    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:08.963950 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:08.963924    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"distinct-2-w5-863b87-kserve-self-signed-certs\""
Nov 02 17:43:08.967004 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:08.966974    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"]
Nov 02 17:43:09.002406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.002368    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/647ee669-772d-4e08-9cf2-73a204ff09fe-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:09.002406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.002406    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/647ee669-772d-4e08-9cf2-73a204ff09fe-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:09.002622 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.002423    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mf5hr\" (UniqueName: \"kubernetes.io/projected/647ee669-772d-4e08-9cf2-73a204ff09fe-kube-api-access-mf5hr\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:09.046317 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.046290    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:09.103931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.103900    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-tmp-dir\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.103931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.103933    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-dshm\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.104196 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.103964    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bffc7364-3157-46e6-8fba-616f128b60cf-tls-certs\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.104196 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.104067    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-home\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.104196 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.104099    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-model-cache\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.104196 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.104158    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cbjks\" (UniqueName: \"kubernetes.io/projected/bffc7364-3157-46e6-8fba-616f128b60cf-kube-api-access-cbjks\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.205882 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.205809    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cbjks\" (UniqueName: \"kubernetes.io/projected/bffc7364-3157-46e6-8fba-616f128b60cf-kube-api-access-cbjks\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.206128 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.206107    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-tmp-dir\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.206231 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.206143    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-dshm\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.206231 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.206188    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bffc7364-3157-46e6-8fba-616f128b60cf-tls-certs\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.206352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.206235    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-home\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.206352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.206260    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-model-cache\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.206864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.206545    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-tmp-dir\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.206864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.206595    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-model-cache\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.206864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.206791    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-home\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.210493 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.210460    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bffc7364-3157-46e6-8fba-616f128b60cf-tls-certs\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.212307 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.212280    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-dshm\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.234843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.233306    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cbjks\" (UniqueName: \"kubernetes.io/projected/bffc7364-3157-46e6-8fba-616f128b60cf-kube-api-access-cbjks\") pod \"distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") " pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.260951 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.260909    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:09.603935 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.602362    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"]
Nov 02 17:43:09.674018 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.673984    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" event={"ID":"bffc7364-3157-46e6-8fba-616f128b60cf","Type":"ContainerStarted","Data":"b2484ce3e10e3880f991bd897ea3087a31547eab5fd9615eaa7df12023f7917b"}
Nov 02 17:43:09.676020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.675991    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2" event={"ID":"647ee669-772d-4e08-9cf2-73a204ff09fe","Type":"ContainerDied","Data":"ac0ecf1782556195cb8115ee390202e1d72fd2f40abf2bf62dec3575d025fab5"}
Nov 02 17:43:09.676178 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.676092    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ac0ecf1782556195cb8115ee390202e1d72fd2f40abf2bf62dec3575d025fab5"
Nov 02 17:43:09.676178 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.676030    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"
Nov 02 17:43:09.678166 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:09.678137    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:10.687795 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:10.686590    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" event={"ID":"bffc7364-3157-46e6-8fba-616f128b60cf","Type":"ContainerStarted","Data":"da563580a735e237934140643b5625df59154f5b6d1ff73d1b0af154dc8aaf39"}
Nov 02 17:43:10.745004 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:10.744934    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podStartSLOduration=2.744912869 podStartE2EDuration="2.744912869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:10.738295171 +0000 UTC m=+807.229358333" watchObservedRunningTime="2026-11-02 17:43:10.744912869 +0000 UTC m=+807.235976032"
Nov 02 17:43:11.262157 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.262123    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:11.493841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.491981    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:11.752018 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.751983    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"]
Nov 02 17:43:11.757316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.757294    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.811839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.795081    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"distinct-2-w1-120394-kserve-self-signed-certs\""
Nov 02 17:43:11.841839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.822536    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"]
Nov 02 17:43:11.885737 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.877253    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-dshm\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.885737 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.885427    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr9ws\" (UniqueName: \"kubernetes.io/projected/57a419b6-5333-4b2c-9f01-051dc82c8707-kube-api-access-lr9ws\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.885737 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.885537    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-model-cache\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.885737 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.885570    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-home\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.886159 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.885784    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a419b6-5333-4b2c-9f01-051dc82c8707-tls-certs\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.886159 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.885810    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-tmp-dir\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.896332 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.887119    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:11.995122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.987482    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-dshm\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.995122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.987556    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr9ws\" (UniqueName: \"kubernetes.io/projected/57a419b6-5333-4b2c-9f01-051dc82c8707-kube-api-access-lr9ws\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.995122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.987651    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-model-cache\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.995122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.987693    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-home\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.995122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.987923    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a419b6-5333-4b2c-9f01-051dc82c8707-tls-certs\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.995122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.987946    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-tmp-dir\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.995122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.988557    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-tmp-dir\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.995122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.989139    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-model-cache\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.995122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.989712    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-home\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.995122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.994443    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-dshm\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:11.995122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:11.994962    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a419b6-5333-4b2c-9f01-051dc82c8707-tls-certs\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:12.029885 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:12.023523    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr9ws\" (UniqueName: \"kubernetes.io/projected/57a419b6-5333-4b2c-9f01-051dc82c8707-kube-api-access-lr9ws\") pod \"distinct-2-w1-120394-kserve-766594cbd8-r7nr9\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") " pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:12.086839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:12.085802    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:12.956897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:12.937584    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"]
Nov 02 17:43:13.511601 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:13.511502    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:13.766849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:13.766748    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" event={"ID":"57a419b6-5333-4b2c-9f01-051dc82c8707","Type":"ContainerStarted","Data":"57939473b6ad2b6cbdafcfe2737dd5cf2ffeefee3affd12a79845d1dc1dea2f7"}
Nov 02 17:43:13.766849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:13.766797    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" event={"ID":"57a419b6-5333-4b2c-9f01-051dc82c8707","Type":"ContainerStarted","Data":"99dcea58b362ef916eb88c98ae6818f61e3f222684ff2c8d11b46cd3b7133bbf"}
Nov 02 17:43:13.814915 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:13.814864    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podStartSLOduration=2.8148502 podStartE2EDuration="2.8148502s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:13.809789392 +0000 UTC m=+810.300852578" watchObservedRunningTime="2026-11-02 17:43:13.8148502 +0000 UTC m=+810.305913361"
Nov 02 17:43:14.086310 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:14.086207    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:15.268604 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:15.268565    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:16.106473 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.106440    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"]
Nov 02 17:43:16.111633 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.111611    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"
Nov 02 17:43:16.132699 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.132665    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"]
Nov 02 17:43:16.139410 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.139369    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:43:16.139567 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.139550    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:43:16.158989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.158950    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c925ba-deb0-4b7f-9542-7ffd5719576d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-887fd0a8-nxk57\" (UID: \"64c925ba-deb0-4b7f-9542-7ffd5719576d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"
Nov 02 17:43:16.159199 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.159175    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gkclk\" (UniqueName: \"kubernetes.io/projected/64c925ba-deb0-4b7f-9542-7ffd5719576d-kube-api-access-gkclk\") pod \"maas-api-revoke-sub-887fd0a8-nxk57\" (UID: \"64c925ba-deb0-4b7f-9542-7ffd5719576d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"
Nov 02 17:43:16.159392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.159368    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c925ba-deb0-4b7f-9542-7ffd5719576d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-887fd0a8-nxk57\" (UID: \"64c925ba-deb0-4b7f-9542-7ffd5719576d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"
Nov 02 17:43:16.174578 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:43:16.174536    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:43:16.260312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.260272    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c925ba-deb0-4b7f-9542-7ffd5719576d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-887fd0a8-nxk57\" (UID: \"64c925ba-deb0-4b7f-9542-7ffd5719576d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"
Nov 02 17:43:16.260486 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.260396    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c925ba-deb0-4b7f-9542-7ffd5719576d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-887fd0a8-nxk57\" (UID: \"64c925ba-deb0-4b7f-9542-7ffd5719576d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"
Nov 02 17:43:16.260530 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.260510    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gkclk\" (UniqueName: \"kubernetes.io/projected/64c925ba-deb0-4b7f-9542-7ffd5719576d-kube-api-access-gkclk\") pod \"maas-api-revoke-sub-887fd0a8-nxk57\" (UID: \"64c925ba-deb0-4b7f-9542-7ffd5719576d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"
Nov 02 17:43:16.260934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.260908    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c925ba-deb0-4b7f-9542-7ffd5719576d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-887fd0a8-nxk57\" (UID: \"64c925ba-deb0-4b7f-9542-7ffd5719576d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"
Nov 02 17:43:16.263004 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.262984    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c925ba-deb0-4b7f-9542-7ffd5719576d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-887fd0a8-nxk57\" (UID: \"64c925ba-deb0-4b7f-9542-7ffd5719576d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"
Nov 02 17:43:16.284741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.284712    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gkclk\" (UniqueName: \"kubernetes.io/projected/64c925ba-deb0-4b7f-9542-7ffd5719576d-kube-api-access-gkclk\") pod \"maas-api-revoke-sub-887fd0a8-nxk57\" (UID: \"64c925ba-deb0-4b7f-9542-7ffd5719576d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"
Nov 02 17:43:16.426507 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.426388    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"
Nov 02 17:43:16.770109 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.770082    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"]
Nov 02 17:43:16.771165 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:43:16.771135    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod64c925ba_deb0_4b7f_9542_7ffd5719576d.slice/crio-1b6c7377fbf8fe7ab2ad7673018d0a0d5a4eac043a05bda82b2f882cfcf19cd3 WatchSource:0}: Error finding container 1b6c7377fbf8fe7ab2ad7673018d0a0d5a4eac043a05bda82b2f882cfcf19cd3: Status 404 returned error can't find the container with id 1b6c7377fbf8fe7ab2ad7673018d0a0d5a4eac043a05bda82b2f882cfcf19cd3
Nov 02 17:43:16.789362 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:16.789332    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57" event={"ID":"64c925ba-deb0-4b7f-9542-7ffd5719576d","Type":"ContainerStarted","Data":"1b6c7377fbf8fe7ab2ad7673018d0a0d5a4eac043a05bda82b2f882cfcf19cd3"}
Nov 02 17:43:17.175367 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:17.175333    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:17.799867 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:17.799827    4456 generic.go:356] "Generic (PLEG): container finished" podID="64c925ba-deb0-4b7f-9542-7ffd5719576d" containerID="f566147f7e53e3e16ae2bc12281b8689cc7650d8a1cb88c8af681363a7b9544d" exitCode=0
Nov 02 17:43:17.800292 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:17.799900    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57" event={"ID":"64c925ba-deb0-4b7f-9542-7ffd5719576d","Type":"ContainerDied","Data":"f566147f7e53e3e16ae2bc12281b8689cc7650d8a1cb88c8af681363a7b9544d"}
Nov 02 17:43:18.976559 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:18.976532    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"
Nov 02 17:43:19.074234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.074141    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"]
Nov 02 17:43:19.074723 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.074396    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54" podUID="5a245c75-8896-4899-9353-7460adf4550a" containerName="istio-proxy" containerID="cri-o://d087b4bc5bcd54eed19435a49f0b6879f565e02ac4f7996dee47d18d3c46d394" gracePeriod=30
Nov 02 17:43:19.100529 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.100491    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/64c925ba-deb0-4b7f-9542-7ffd5719576d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c925ba-deb0-4b7f-9542-7ffd5719576d-maas-api-xks-ca\") pod \"64c925ba-deb0-4b7f-9542-7ffd5719576d\" (UID: \"64c925ba-deb0-4b7f-9542-7ffd5719576d\") "
Nov 02 17:43:19.100699 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.100575    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/64c925ba-deb0-4b7f-9542-7ffd5719576d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c925ba-deb0-4b7f-9542-7ffd5719576d-maas-api-openshift-ca\") pod \"64c925ba-deb0-4b7f-9542-7ffd5719576d\" (UID: \"64c925ba-deb0-4b7f-9542-7ffd5719576d\") "
Nov 02 17:43:19.100699 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.100624    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/64c925ba-deb0-4b7f-9542-7ffd5719576d-kube-api-access-gkclk\" (UniqueName: \"kubernetes.io/projected/64c925ba-deb0-4b7f-9542-7ffd5719576d-kube-api-access-gkclk\") pod \"64c925ba-deb0-4b7f-9542-7ffd5719576d\" (UID: \"64c925ba-deb0-4b7f-9542-7ffd5719576d\") "
Nov 02 17:43:19.101513 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.101404    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/64c925ba-deb0-4b7f-9542-7ffd5719576d-maas-api-openshift-ca" pod "64c925ba-deb0-4b7f-9542-7ffd5719576d" (UID: "64c925ba-deb0-4b7f-9542-7ffd5719576d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:19.103703 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.103665    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/64c925ba-deb0-4b7f-9542-7ffd5719576d-kube-api-access-gkclk" pod "64c925ba-deb0-4b7f-9542-7ffd5719576d" (UID: "64c925ba-deb0-4b7f-9542-7ffd5719576d"). InnerVolumeSpecName "kube-api-access-gkclk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:19.104411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.104385    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/64c925ba-deb0-4b7f-9542-7ffd5719576d-maas-api-xks-ca" pod "64c925ba-deb0-4b7f-9542-7ffd5719576d" (UID: "64c925ba-deb0-4b7f-9542-7ffd5719576d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:43:19.202959 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.202923    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/64c925ba-deb0-4b7f-9542-7ffd5719576d-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:19.202959 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.202953    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/64c925ba-deb0-4b7f-9542-7ffd5719576d-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:19.202959 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.202967    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gkclk\" (UniqueName: \"kubernetes.io/projected/64c925ba-deb0-4b7f-9542-7ffd5719576d-kube-api-access-gkclk\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:19.245038 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.245005    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:19.262120 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.262068    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:19.290422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.290383    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:19.814567 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.814539    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"
Nov 02 17:43:19.814567 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.814549    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57" event={"ID":"64c925ba-deb0-4b7f-9542-7ffd5719576d","Type":"ContainerDied","Data":"1b6c7377fbf8fe7ab2ad7673018d0a0d5a4eac043a05bda82b2f882cfcf19cd3"}
Nov 02 17:43:19.814853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.814604    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1b6c7377fbf8fe7ab2ad7673018d0a0d5a4eac043a05bda82b2f882cfcf19cd3"
Nov 02 17:43:19.826742 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:19.826716    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:21.159162 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:21.159106    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:22.086654 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:22.086626    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:22.100681 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:22.100651    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:22.848806 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:22.848780    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:23.066645 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:23.066602    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:24.367871 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.367835    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:43:24.478473 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.478445    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5a245c75-8896-4899-9353-7460adf4550a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a245c75-8896-4899-9353-7460adf4550a-istiod-ca-cert\") pod \"5a245c75-8896-4899-9353-7460adf4550a\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") "
Nov 02 17:43:24.478638 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.478513    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-workload-certs\") pod \"5a245c75-8896-4899-9353-7460adf4550a\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") "
Nov 02 17:43:24.478638 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.478593    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5a245c75-8896-4899-9353-7460adf4550a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a245c75-8896-4899-9353-7460adf4550a-istio-podinfo\") pod \"5a245c75-8896-4899-9353-7460adf4550a\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") "
Nov 02 17:43:24.478736 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.478639    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-istio-data\") pod \"5a245c75-8896-4899-9353-7460adf4550a\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") "
Nov 02 17:43:24.478736 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.478669    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a245c75-8896-4899-9353-7460adf4550a-istio-token\" (UniqueName: \"kubernetes.io/projected/5a245c75-8896-4899-9353-7460adf4550a-istio-token\") pod \"5a245c75-8896-4899-9353-7460adf4550a\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") "
Nov 02 17:43:24.478736 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.478688    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-workload-socket\") pod \"5a245c75-8896-4899-9353-7460adf4550a\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") "
Nov 02 17:43:24.478892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.478769    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-credential-socket\") pod \"5a245c75-8896-4899-9353-7460adf4550a\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") "
Nov 02 17:43:24.478892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.478859    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5a245c75-8896-4899-9353-7460adf4550a-kube-api-access-mxlhh\" (UniqueName: \"kubernetes.io/projected/5a245c75-8896-4899-9353-7460adf4550a-kube-api-access-mxlhh\") pod \"5a245c75-8896-4899-9353-7460adf4550a\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") "
Nov 02 17:43:24.479061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.478881    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-workload-certs" pod "5a245c75-8896-4899-9353-7460adf4550a" (UID: "5a245c75-8896-4899-9353-7460adf4550a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:43:24.479061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.478907    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5a245c75-8896-4899-9353-7460adf4550a-istiod-ca-cert" pod "5a245c75-8896-4899-9353-7460adf4550a" (UID: "5a245c75-8896-4899-9353-7460adf4550a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:24.479061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.478917    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-istio-envoy\") pod \"5a245c75-8896-4899-9353-7460adf4550a\" (UID: \"5a245c75-8896-4899-9353-7460adf4550a\") "
Nov 02 17:43:24.479202 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.479072    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-istio-data" pod "5a245c75-8896-4899-9353-7460adf4550a" (UID: "5a245c75-8896-4899-9353-7460adf4550a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:43:24.479202 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.479140    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-workload-socket" pod "5a245c75-8896-4899-9353-7460adf4550a" (UID: "5a245c75-8896-4899-9353-7460adf4550a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:43:24.479317 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.479242    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-credential-socket" pod "5a245c75-8896-4899-9353-7460adf4550a" (UID: "5a245c75-8896-4899-9353-7460adf4550a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:43:24.479671 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.479590    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:24.479671 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.479612    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:24.479671 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.479621    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:24.479671 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.479630    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5a245c75-8896-4899-9353-7460adf4550a-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:24.479671 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.479638    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:24.481482 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.481461    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-istio-envoy" pod "5a245c75-8896-4899-9353-7460adf4550a" (UID: "5a245c75-8896-4899-9353-7460adf4550a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:43:24.481763 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.481738    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5a245c75-8896-4899-9353-7460adf4550a-istio-podinfo" pod "5a245c75-8896-4899-9353-7460adf4550a" (UID: "5a245c75-8896-4899-9353-7460adf4550a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:43:24.481903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.481784    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a245c75-8896-4899-9353-7460adf4550a-istio-token" pod "5a245c75-8896-4899-9353-7460adf4550a" (UID: "5a245c75-8896-4899-9353-7460adf4550a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:24.481903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.481845    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5a245c75-8896-4899-9353-7460adf4550a-kube-api-access-mxlhh" pod "5a245c75-8896-4899-9353-7460adf4550a" (UID: "5a245c75-8896-4899-9353-7460adf4550a"). InnerVolumeSpecName "kube-api-access-mxlhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:24.580557 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.580518    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5a245c75-8896-4899-9353-7460adf4550a-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:24.580557 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.580550    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5a245c75-8896-4899-9353-7460adf4550a-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:24.580557 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.580564    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mxlhh\" (UniqueName: \"kubernetes.io/projected/5a245c75-8896-4899-9353-7460adf4550a-kube-api-access-mxlhh\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:24.580804 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.580577    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5a245c75-8896-4899-9353-7460adf4550a-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:24.854212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.854177    4456 generic.go:356] "Generic (PLEG): container finished" podID="5a245c75-8896-4899-9353-7460adf4550a" containerID="d087b4bc5bcd54eed19435a49f0b6879f565e02ac4f7996dee47d18d3c46d394" exitCode=0
Nov 02 17:43:24.854394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.854229    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"
Nov 02 17:43:24.854394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.854266    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54" event={"ID":"5a245c75-8896-4899-9353-7460adf4550a","Type":"ContainerDied","Data":"d087b4bc5bcd54eed19435a49f0b6879f565e02ac4f7996dee47d18d3c46d394"}
Nov 02 17:43:24.854394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.854334    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54" event={"ID":"5a245c75-8896-4899-9353-7460adf4550a","Type":"ContainerDied","Data":"f760f0085515067c3b0f4fce03bb3e1caddc6d00943d5ccd35e9472c8d1ee91d"}
Nov 02 17:43:24.854394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.854362    4456 scope.go:122] "RemoveContainer" containerID="d087b4bc5bcd54eed19435a49f0b6879f565e02ac4f7996dee47d18d3c46d394"
Nov 02 17:43:24.865954 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.865938    4456 scope.go:122] "RemoveContainer" containerID="d087b4bc5bcd54eed19435a49f0b6879f565e02ac4f7996dee47d18d3c46d394"
Nov 02 17:43:24.866220 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:43:24.866206    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d087b4bc5bcd54eed19435a49f0b6879f565e02ac4f7996dee47d18d3c46d394\": container with ID starting with d087b4bc5bcd54eed19435a49f0b6879f565e02ac4f7996dee47d18d3c46d394 not found: ID does not exist" containerID="d087b4bc5bcd54eed19435a49f0b6879f565e02ac4f7996dee47d18d3c46d394"
Nov 02 17:43:24.866260 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.866231    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d087b4bc5bcd54eed19435a49f0b6879f565e02ac4f7996dee47d18d3c46d394"} err="failed to get container status \"d087b4bc5bcd54eed19435a49f0b6879f565e02ac4f7996dee47d18d3c46d394\": rpc error: code = NotFound desc = could not find container \"d087b4bc5bcd54eed19435a49f0b6879f565e02ac4f7996dee47d18d3c46d394\": container with ID starting with d087b4bc5bcd54eed19435a49f0b6879f565e02ac4f7996dee47d18d3c46d394 not found: ID does not exist"
Nov 02 17:43:24.916626 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.916600    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"]
Nov 02 17:43:24.929336 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.929302    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vlw54"]
Nov 02 17:43:24.952753 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:24.952719    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:25.615056 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:25.615021    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5a245c75-8896-4899-9353-7460adf4550a" path="/var/lib/kubelet/pods/5a245c75-8896-4899-9353-7460adf4550a/volumes"
Nov 02 17:43:26.927784 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:26.927750    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"]
Nov 02 17:43:26.928665 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:26.928643    4456 cpu_manager.go:414] "containerMap: removing container" podUID="64c925ba-deb0-4b7f-9542-7ffd5719576d" containerName="revoke-keys"
Nov 02 17:43:26.928715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:26.928669    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="64c925ba-deb0-4b7f-9542-7ffd5719576d" containerName="revoke-keys"
Nov 02 17:43:26.928715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:26.928696    4456 cpu_manager.go:414] "containerMap: removing container" podUID="5a245c75-8896-4899-9353-7460adf4550a" containerName="istio-proxy"
Nov 02 17:43:26.928715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:26.928703    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5a245c75-8896-4899-9353-7460adf4550a" containerName="istio-proxy"
Nov 02 17:43:26.928903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:26.928889    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="64c925ba-deb0-4b7f-9542-7ffd5719576d" containerName="revoke-keys"
Nov 02 17:43:26.928948 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:26.928911    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5a245c75-8896-4899-9353-7460adf4550a" containerName="istio-proxy"
Nov 02 17:43:26.934683 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:26.934664    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"
Nov 02 17:43:26.958397 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:26.958365    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:43:26.959259 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:26.959117    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:43:26.960705 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:26.960677    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:26.974656 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:26.974603    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"]
Nov 02 17:43:27.015728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:27.015690    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pprfp\" (UniqueName: \"kubernetes.io/projected/9b172da9-d254-45eb-aa9f-16e72704dd06-kube-api-access-pprfp\") pod \"maas-api-revoke-sub-81dc9fb1-wqtt9\" (UID: \"9b172da9-d254-45eb-aa9f-16e72704dd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"
Nov 02 17:43:27.015958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:27.015864    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b172da9-d254-45eb-aa9f-16e72704dd06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81dc9fb1-wqtt9\" (UID: \"9b172da9-d254-45eb-aa9f-16e72704dd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"
Nov 02 17:43:27.016023 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:27.015977    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b172da9-d254-45eb-aa9f-16e72704dd06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81dc9fb1-wqtt9\" (UID: \"9b172da9-d254-45eb-aa9f-16e72704dd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"
Nov 02 17:43:27.117147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:27.117103    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b172da9-d254-45eb-aa9f-16e72704dd06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81dc9fb1-wqtt9\" (UID: \"9b172da9-d254-45eb-aa9f-16e72704dd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"
Nov 02 17:43:27.117322 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:27.117263    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pprfp\" (UniqueName: \"kubernetes.io/projected/9b172da9-d254-45eb-aa9f-16e72704dd06-kube-api-access-pprfp\") pod \"maas-api-revoke-sub-81dc9fb1-wqtt9\" (UID: \"9b172da9-d254-45eb-aa9f-16e72704dd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"
Nov 02 17:43:27.117400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:27.117386    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b172da9-d254-45eb-aa9f-16e72704dd06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81dc9fb1-wqtt9\" (UID: \"9b172da9-d254-45eb-aa9f-16e72704dd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"
Nov 02 17:43:27.118045 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:27.118021    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b172da9-d254-45eb-aa9f-16e72704dd06-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-81dc9fb1-wqtt9\" (UID: \"9b172da9-d254-45eb-aa9f-16e72704dd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"
Nov 02 17:43:27.119715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:27.119688    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b172da9-d254-45eb-aa9f-16e72704dd06-maas-api-xks-ca\") pod \"maas-api-revoke-sub-81dc9fb1-wqtt9\" (UID: \"9b172da9-d254-45eb-aa9f-16e72704dd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"
Nov 02 17:43:27.142442 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:27.142418    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pprfp\" (UniqueName: \"kubernetes.io/projected/9b172da9-d254-45eb-aa9f-16e72704dd06-kube-api-access-pprfp\") pod \"maas-api-revoke-sub-81dc9fb1-wqtt9\" (UID: \"9b172da9-d254-45eb-aa9f-16e72704dd06\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"
Nov 02 17:43:27.268212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:27.268170    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"
Nov 02 17:43:27.577489 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:27.577467    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"]
Nov 02 17:43:27.577975 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:43:27.577951    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b172da9_d254_45eb_aa9f_16e72704dd06.slice/crio-7d5d4d7a6e5d2bafa37a0d3bb92f546b82b28c9fe758fb119867b0c9d32a7db7 WatchSource:0}: Error finding container 7d5d4d7a6e5d2bafa37a0d3bb92f546b82b28c9fe758fb119867b0c9d32a7db7: Status 404 returned error can't find the container with id 7d5d4d7a6e5d2bafa37a0d3bb92f546b82b28c9fe758fb119867b0c9d32a7db7
Nov 02 17:43:27.876214 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:27.876124    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9" event={"ID":"9b172da9-d254-45eb-aa9f-16e72704dd06","Type":"ContainerStarted","Data":"7d5d4d7a6e5d2bafa37a0d3bb92f546b82b28c9fe758fb119867b0c9d32a7db7"}
Nov 02 17:43:28.769398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:28.769285    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:28.888577 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:28.888545    4456 generic.go:356] "Generic (PLEG): container finished" podID="9b172da9-d254-45eb-aa9f-16e72704dd06" containerID="a2575df33c8bc59b1996d3ad98250e29beef5e1a5d7ad2610ca1f7306edee51a" exitCode=0
Nov 02 17:43:28.888764 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:28.888594    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9" event={"ID":"9b172da9-d254-45eb-aa9f-16e72704dd06","Type":"ContainerDied","Data":"a2575df33c8bc59b1996d3ad98250e29beef5e1a5d7ad2610ca1f7306edee51a"}
Nov 02 17:43:30.053047 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:30.053022    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"
Nov 02 17:43:30.162171 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:30.162080    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b172da9-d254-45eb-aa9f-16e72704dd06-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b172da9-d254-45eb-aa9f-16e72704dd06-maas-api-openshift-ca\") pod \"9b172da9-d254-45eb-aa9f-16e72704dd06\" (UID: \"9b172da9-d254-45eb-aa9f-16e72704dd06\") "
Nov 02 17:43:30.162171 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:30.162155    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b172da9-d254-45eb-aa9f-16e72704dd06-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b172da9-d254-45eb-aa9f-16e72704dd06-maas-api-xks-ca\") pod \"9b172da9-d254-45eb-aa9f-16e72704dd06\" (UID: \"9b172da9-d254-45eb-aa9f-16e72704dd06\") "
Nov 02 17:43:30.162375 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:30.162310    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b172da9-d254-45eb-aa9f-16e72704dd06-kube-api-access-pprfp\" (UniqueName: \"kubernetes.io/projected/9b172da9-d254-45eb-aa9f-16e72704dd06-kube-api-access-pprfp\") pod \"9b172da9-d254-45eb-aa9f-16e72704dd06\" (UID: \"9b172da9-d254-45eb-aa9f-16e72704dd06\") "
Nov 02 17:43:30.162461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:30.162440    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b172da9-d254-45eb-aa9f-16e72704dd06-maas-api-openshift-ca" pod "9b172da9-d254-45eb-aa9f-16e72704dd06" (UID: "9b172da9-d254-45eb-aa9f-16e72704dd06"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:30.162909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:30.162805    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b172da9-d254-45eb-aa9f-16e72704dd06-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:30.164343 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:30.164314    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b172da9-d254-45eb-aa9f-16e72704dd06-maas-api-xks-ca" pod "9b172da9-d254-45eb-aa9f-16e72704dd06" (UID: "9b172da9-d254-45eb-aa9f-16e72704dd06"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:43:30.164477 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:30.164459    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b172da9-d254-45eb-aa9f-16e72704dd06-kube-api-access-pprfp" pod "9b172da9-d254-45eb-aa9f-16e72704dd06" (UID: "9b172da9-d254-45eb-aa9f-16e72704dd06"). InnerVolumeSpecName "kube-api-access-pprfp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:30.264431 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:30.264398    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pprfp\" (UniqueName: \"kubernetes.io/projected/9b172da9-d254-45eb-aa9f-16e72704dd06-kube-api-access-pprfp\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:30.264431 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:30.264426    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b172da9-d254-45eb-aa9f-16e72704dd06-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:30.688382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:30.688284    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:30.905738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:30.905707    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"
Nov 02 17:43:30.905945 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:30.905709    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9" event={"ID":"9b172da9-d254-45eb-aa9f-16e72704dd06","Type":"ContainerDied","Data":"7d5d4d7a6e5d2bafa37a0d3bb92f546b82b28c9fe758fb119867b0c9d32a7db7"}
Nov 02 17:43:30.905945 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:30.905846    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7d5d4d7a6e5d2bafa37a0d3bb92f546b82b28c9fe758fb119867b0c9d32a7db7"
Nov 02 17:43:31.218497 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.218457    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"]
Nov 02 17:43:31.219408 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.219384    4456 cpu_manager.go:414] "containerMap: removing container" podUID="9b172da9-d254-45eb-aa9f-16e72704dd06" containerName="revoke-keys"
Nov 02 17:43:31.219514 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.219410    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b172da9-d254-45eb-aa9f-16e72704dd06" containerName="revoke-keys"
Nov 02 17:43:31.219620 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.219605    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b172da9-d254-45eb-aa9f-16e72704dd06" containerName="revoke-keys"
Nov 02 17:43:31.223931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.223910    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.288946 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.288912    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"]
Nov 02 17:43:31.379955 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.379917    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fe85b3-015f-4145-abba-9d31009c6229-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.380145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.380034    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.380145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.380096    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.380258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.380148    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fe85b3-015f-4145-abba-9d31009c6229-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.380258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.380196    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-64dlm\" (UniqueName: \"kubernetes.io/projected/54fe85b3-015f-4145-abba-9d31009c6229-kube-api-access-64dlm\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.380356 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.380295    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fe85b3-015f-4145-abba-9d31009c6229-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.380356 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.380327    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.380470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.380446    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.380523 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.380480    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.482296 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.482190    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.482296 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.482242    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.482519 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.482302    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fe85b3-015f-4145-abba-9d31009c6229-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.482519 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.482378    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.482519 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.482459    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.482661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.482528    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fe85b3-015f-4145-abba-9d31009c6229-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.482661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.482552    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-64dlm\" (UniqueName: \"kubernetes.io/projected/54fe85b3-015f-4145-abba-9d31009c6229-kube-api-access-64dlm\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.482758 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.482662    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fe85b3-015f-4145-abba-9d31009c6229-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.482758 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.482709    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.484215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.483527    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-workload-certs\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.484215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.483564    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-workload-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.484215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.484173    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-credential-socket\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.484437 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.484413    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fe85b3-015f-4145-abba-9d31009c6229-istiod-ca-cert\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.484696 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.484676    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-istio-data\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.486978 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.486926    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-istio-envoy\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.487562 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.487537    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fe85b3-015f-4145-abba-9d31009c6229-istio-podinfo\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.515658 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.515625    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fe85b3-015f-4145-abba-9d31009c6229-istio-token\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.518187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.518164    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-64dlm\" (UniqueName: \"kubernetes.io/projected/54fe85b3-015f-4145-abba-9d31009c6229-kube-api-access-64dlm\") pod \"e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") " pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.540393 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.540369    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:31.888298 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.888268    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"]
Nov 02 17:43:31.889475 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:43:31.889448    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod54fe85b3_015f_4145_abba_9d31009c6229.slice/crio-b9e36cb2167c1643fbb7a7a55bc5acc2dd22994a047c1b91138490b2b36186be WatchSource:0}: Error finding container b9e36cb2167c1643fbb7a7a55bc5acc2dd22994a047c1b91138490b2b36186be: Status 404 returned error can't find the container with id b9e36cb2167c1643fbb7a7a55bc5acc2dd22994a047c1b91138490b2b36186be
Nov 02 17:43:31.897301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.897264    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:31.897408 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.897333    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:31.897408 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.897365    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:31.915888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:31.915851    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" event={"ID":"54fe85b3-015f-4145-abba-9d31009c6229","Type":"ContainerStarted","Data":"b9e36cb2167c1643fbb7a7a55bc5acc2dd22994a047c1b91138490b2b36186be"}
Nov 02 17:43:32.926146 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:32.926108    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" event={"ID":"54fe85b3-015f-4145-abba-9d31009c6229","Type":"ContainerStarted","Data":"b93fc8271576e0f58e400c54457f3aa6226f1bed6d4e87a7f2447ca8db7e58fe"}
Nov 02 17:43:32.977797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:32.977763    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:33.045838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:33.034493    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" podStartSLOduration=2.03447207 podStartE2EDuration="2.03447207s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:32.997339262 +0000 UTC m=+829.488402454" watchObservedRunningTime="2026-11-02 17:43:33.03447207 +0000 UTC m=+829.525535236"
Nov 02 17:43:33.540524 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:33.540487    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:33.542056 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:33.542031    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:33.542227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:33.542081    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 17:43:33.728029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:33.727997    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:34.541587 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:34.541542    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:34.541998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:34.541618    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 17:43:35.124843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.124557    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:35.455038 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.455003    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"]
Nov 02 17:43:35.467447 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.467416    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"
Nov 02 17:43:35.477919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.477891    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"]
Nov 02 17:43:35.499759 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.499728    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:43:35.520035 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.519088    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:43:35.540738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.540706    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:35.540920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.540778    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 17:43:35.690841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.678930    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59320ce-836a-422e-9717-61df0ec341d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2585040a-p8z5j\" (UID: \"f59320ce-836a-422e-9717-61df0ec341d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"
Nov 02 17:43:35.691468 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.691439    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cs4kh\" (UniqueName: \"kubernetes.io/projected/f59320ce-836a-422e-9717-61df0ec341d0-kube-api-access-cs4kh\") pod \"maas-api-revoke-sub-2585040a-p8z5j\" (UID: \"f59320ce-836a-422e-9717-61df0ec341d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"
Nov 02 17:43:35.691605 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.691590    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59320ce-836a-422e-9717-61df0ec341d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2585040a-p8z5j\" (UID: \"f59320ce-836a-422e-9717-61df0ec341d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"
Nov 02 17:43:35.803697 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.803594    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59320ce-836a-422e-9717-61df0ec341d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2585040a-p8z5j\" (UID: \"f59320ce-836a-422e-9717-61df0ec341d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"
Nov 02 17:43:35.805692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.805666    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cs4kh\" (UniqueName: \"kubernetes.io/projected/f59320ce-836a-422e-9717-61df0ec341d0-kube-api-access-cs4kh\") pod \"maas-api-revoke-sub-2585040a-p8z5j\" (UID: \"f59320ce-836a-422e-9717-61df0ec341d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"
Nov 02 17:43:35.806202 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.806182    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59320ce-836a-422e-9717-61df0ec341d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2585040a-p8z5j\" (UID: \"f59320ce-836a-422e-9717-61df0ec341d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"
Nov 02 17:43:35.807053 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.805481    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59320ce-836a-422e-9717-61df0ec341d0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2585040a-p8z5j\" (UID: \"f59320ce-836a-422e-9717-61df0ec341d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"
Nov 02 17:43:35.846911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.846880    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59320ce-836a-422e-9717-61df0ec341d0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2585040a-p8z5j\" (UID: \"f59320ce-836a-422e-9717-61df0ec341d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"
Nov 02 17:43:35.878633 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:35.878597    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cs4kh\" (UniqueName: \"kubernetes.io/projected/f59320ce-836a-422e-9717-61df0ec341d0-kube-api-access-cs4kh\") pod \"maas-api-revoke-sub-2585040a-p8z5j\" (UID: \"f59320ce-836a-422e-9717-61df0ec341d0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"
Nov 02 17:43:36.125152 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:36.125054    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"
Nov 02 17:43:36.717295 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:36.706889    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:36.717295 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:36.706960    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 17:43:36.844507 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:36.844463    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:36.960621 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:36.947958    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 17:43:36.972318 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:36.972274    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:36.978121 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:36.977029    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 17:43:37.110913 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.110338    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"]
Nov 02 17:43:37.135841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.126904    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 17:43:37.238919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.230508    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j" event={"ID":"f59320ce-836a-422e-9717-61df0ec341d0","Type":"ContainerStarted","Data":"0f1d9deae0372fb17fb5fd5f427cd48091e4b0eec16bd2ec685797403efeda01"}
Nov 02 17:43:37.543843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.543073    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:37.543843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.543140    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 17:43:37.726842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.723905    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 17:43:37.726842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.725365    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:43:37.756841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.753773    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:37.842741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.842692    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:37.852848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.849920    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:37.852848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.850721    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:37.951975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.951917    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:37.958880 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.956600    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:43:38.022842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.977968    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:43:38.022842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.978516    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:43:38.022842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.979688    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:38.022842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:37.987320    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:43:38.146842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.146769    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 17:43:38.147177 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.147149    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:43:38.544843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.543450    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:38.544843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.543525    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 17:43:38.643837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.642951    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:38.807918 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.806185    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:43:38.807918 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.807007    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:43:38.808889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.808856    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:43:38.809172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.809150    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 17:43:38.809459 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.809437    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:43:38.809526 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.809489    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:43:38.849952 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.849903    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:38.879849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.850256    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:38.882075 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.881699    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:38.979847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.961093    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:39.006869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.995545    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:39.006869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:38.997935    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:43:39.093970 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:39.093869    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j" event={"ID":"f59320ce-836a-422e-9717-61df0ec341d0","Type":"ContainerStarted","Data":"4db660d01066fd539442dd27fc279e629403a01712229dc39f65b23ded3509d9"}
Nov 02 17:43:39.266973 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:39.154108    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:39.880844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:39.793020    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j" podStartSLOduration=4.792996143 podStartE2EDuration="4.792996143s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:35 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:39.350368338 +0000 UTC m=+835.841431528" watchObservedRunningTime="2026-11-02 17:43:39.792996143 +0000 UTC m=+836.284059305"
Nov 02 17:43:39.880844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:39.821616    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:39.880844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:39.821676    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 17:43:39.880844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:39.851058    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 17:43:39.912896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:39.910231    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:40.025985 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:39.998635    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:40.159931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.042289    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 17:43:40.291843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.286866    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 17:43:40.383556 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.383505    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 17:43:40.393368 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.393312    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:43:40.484627 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.396956    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:43:40.484923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.484887    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj" podUID="22b39708-251c-44ad-9b48-2f3c71519131" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.53:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:40.513844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.404531    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:43:40.514628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.286182    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:40.553180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.435070    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Nov 02 17:43:40.553180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.435136    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:43:40.553180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.532188    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:40.553180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.436463    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:43:40.553180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.532247    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96" podUID="3891e06c-0c53-421c-b379-c38efca65905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:40.553652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.435101    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded"
Nov 02 17:43:40.600971 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.558186    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:40.609841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.601191    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 17:43:40.646970 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.621648    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:40.800123 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.800022    4456 generic.go:356] "Generic (PLEG): container finished" podID="f59320ce-836a-422e-9717-61df0ec341d0" containerID="4db660d01066fd539442dd27fc279e629403a01712229dc39f65b23ded3509d9" exitCode=0
Nov 02 17:43:40.803075 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.801701    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j" event={"ID":"f59320ce-836a-422e-9717-61df0ec341d0","Type":"ContainerDied","Data":"4db660d01066fd539442dd27fc279e629403a01712229dc39f65b23ded3509d9"}
Nov 02 17:43:40.845952 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.845916    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:43:40.870957 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.870908    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh" podUID="426b74e4-82bb-46fc-b57f-f6ae93bcbefc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.64:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:40.871274 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.871247    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:40.999842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.974031    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:41.013976 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:40.974110    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 17:43:41.244316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:41.157150    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:41.244316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:41.158528    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:43:41.244316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:41.158572    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq" podUID="c1d9a093-015c-498b-8808-7939c01499da" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.68:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:41.244316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:41.165033    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:41.244316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:41.197960    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:43:41.244316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:41.198013    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:41.339182 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:41.339132    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:41.389633 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:41.389587    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:41.421941 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:41.415991    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Nov 02 17:43:41.577008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:41.539083    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:41.577008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:41.539438    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:43:41.577008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:41.540692    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:41.577008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:41.540729    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 17:43:41.577008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:41.561786    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:41.892053 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:41.891940    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:42.056859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:42.042470    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:42.174898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:42.161160    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:42.174898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:42.172527    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:42.354854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:42.341916    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:42.398147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:42.394451    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:42.408915 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:42.398895    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:43:42.408915 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:42.398945    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:42.542254 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:42.542217    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:42.542662 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:42.542636    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 17:43:42.543230 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:42.542565    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:43:42.543619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:42.542532    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:42.572881 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:42.572840    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 17:43:42.772001 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:42.771961    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:43:42.772279 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:42.772251    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9" podUID="67472e82-1ea4-420c-81cd-91952f3d0bf7" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.72:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:42.959664 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:42.959623    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:43.112785 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.112073    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 17:43:43.147565 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.147257    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:43.184855 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.184122    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:43.184855 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.184316    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:43.206314 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.205056    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:43.307875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.307844    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"
Nov 02 17:43:43.352993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.352948    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:43.396851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.369094    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f59320ce-836a-422e-9717-61df0ec341d0-kube-api-access-cs4kh\" (UniqueName: \"kubernetes.io/projected/f59320ce-836a-422e-9717-61df0ec341d0-kube-api-access-cs4kh\") pod \"f59320ce-836a-422e-9717-61df0ec341d0\" (UID: \"f59320ce-836a-422e-9717-61df0ec341d0\") "
Nov 02 17:43:43.397240 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.397209    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f59320ce-836a-422e-9717-61df0ec341d0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59320ce-836a-422e-9717-61df0ec341d0-maas-api-openshift-ca\") pod \"f59320ce-836a-422e-9717-61df0ec341d0\" (UID: \"f59320ce-836a-422e-9717-61df0ec341d0\") "
Nov 02 17:43:43.422584 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.422545    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f59320ce-836a-422e-9717-61df0ec341d0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59320ce-836a-422e-9717-61df0ec341d0-maas-api-xks-ca\") pod \"f59320ce-836a-422e-9717-61df0ec341d0\" (UID: \"f59320ce-836a-422e-9717-61df0ec341d0\") "
Nov 02 17:43:43.423599 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.409347    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f59320ce-836a-422e-9717-61df0ec341d0-maas-api-openshift-ca" pod "f59320ce-836a-422e-9717-61df0ec341d0" (UID: "f59320ce-836a-422e-9717-61df0ec341d0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:43.423755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.409432    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:43:43.423948 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.423917    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs" podUID="35675ac4-845c-4700-95c7-9f3ba72bd20a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.70:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:43.424091 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.394739    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:43.430465 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.409875    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f59320ce-836a-422e-9717-61df0ec341d0-kube-api-access-cs4kh" pod "f59320ce-836a-422e-9717-61df0ec341d0" (UID: "f59320ce-836a-422e-9717-61df0ec341d0"). InnerVolumeSpecName "kube-api-access-cs4kh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:43.455079 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.454347    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f59320ce-836a-422e-9717-61df0ec341d0-maas-api-xks-ca" pod "f59320ce-836a-422e-9717-61df0ec341d0" (UID: "f59320ce-836a-422e-9717-61df0ec341d0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:43:43.546595 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.543069    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cs4kh\" (UniqueName: \"kubernetes.io/projected/f59320ce-836a-422e-9717-61df0ec341d0-kube-api-access-cs4kh\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:43.546595 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.543105    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f59320ce-836a-422e-9717-61df0ec341d0-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:43.546595 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.543119    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f59320ce-836a-422e-9717-61df0ec341d0-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:43.546595 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.544011    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:43.551663 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.551340    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:43:43.594877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.589634    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 17:43:43.668412 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.668100    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:43.669129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.669071    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:43.669256 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.669136    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 17:43:43.944859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.941906    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:43:43.956337 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.956308    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:43:43.964598 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.964548    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:43:43.970463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:43.970436    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:43:44.013839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:44.009288    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:43:44.023859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:44.023159    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:43:44.027437 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:44.027414    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:43:44.061268 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:44.061233    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:43:44.095922 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:44.095894    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:43:44.144672 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:44.141971    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:43:44.163699 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:44.163657    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j" event={"ID":"f59320ce-836a-422e-9717-61df0ec341d0","Type":"ContainerDied","Data":"0f1d9deae0372fb17fb5fd5f427cd48091e4b0eec16bd2ec685797403efeda01"}
Nov 02 17:43:44.163909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:44.163724    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f1d9deae0372fb17fb5fd5f427cd48091e4b0eec16bd2ec685797403efeda01"
Nov 02 17:43:44.163909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:44.163801    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"
Nov 02 17:43:44.540874 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:44.540775    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused" start-of-body=
Nov 02 17:43:44.541152 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:44.541124    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": dial tcp 10.134.0.80:15021: connect: connection refused"
Nov 02 17:43:45.207326 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:45.207233    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:45.619993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:45.619902    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:45.621066 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:45.621048    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:45.621983 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:45.621961    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:43:46.172341 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:43:46.172301    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:43:46.319713 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.319664    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"]
Nov 02 17:43:46.321071 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.321030    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f59320ce-836a-422e-9717-61df0ec341d0" containerName="revoke-keys"
Nov 02 17:43:46.321213 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.321102    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f59320ce-836a-422e-9717-61df0ec341d0" containerName="revoke-keys"
Nov 02 17:43:46.321361 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.321344    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f59320ce-836a-422e-9717-61df0ec341d0" containerName="revoke-keys"
Nov 02 17:43:46.327569 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.327543    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.357845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.357479    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"]
Nov 02 17:43:46.435998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.432745    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.435998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.432928    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.435998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.432970    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.435998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.433093    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.435998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.433121    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.435998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.433174    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.435998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.433223    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.435998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.433250    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.435998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.433300    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qncn2\" (UniqueName: \"kubernetes.io/projected/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-kube-api-access-qncn2\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.534869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.534788    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.535247 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.535220    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.535364 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.535344    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.535490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.535472    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.535551 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.535507    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.535551 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.535536    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.535661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.535583    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.535661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.535607    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.535758 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.535661    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qncn2\" (UniqueName: \"kubernetes.io/projected/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-kube-api-access-qncn2\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.535758 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.535675    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.536019 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.535993    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.536140 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.536088    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.536362 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.536339    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.537069 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.537039    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.538391 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.538369    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.539087 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.539063    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.561400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.561344    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.563518 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.563213    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qncn2\" (UniqueName: \"kubernetes.io/projected/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-kube-api-access-qncn2\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:46.661238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:46.661202    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:47.035985 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:47.035958    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"]
Nov 02 17:43:47.036860 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:43:47.036832    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddc7a0bba_95b4_4325_8775_a0a73ff1d67a.slice/crio-f8b30213b2383eb24cd57f5c36f35cf41d38efe10771bd97060fc122460d90da WatchSource:0}: Error finding container f8b30213b2383eb24cd57f5c36f35cf41d38efe10771bd97060fc122460d90da: Status 404 returned error can't find the container with id f8b30213b2383eb24cd57f5c36f35cf41d38efe10771bd97060fc122460d90da
Nov 02 17:43:47.040187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:47.040148    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:47.040287 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:47.040222    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:47.040287 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:47.040250    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:43:47.205670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:47.205630    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:47.211892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:47.211863    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b" event={"ID":"dc7a0bba-95b4-4325-8775-a0a73ff1d67a","Type":"ContainerStarted","Data":"1f3a978598430e7c27106ecf598ecc76acc2a9ae638e68714d12e4f949cf8947"}
Nov 02 17:43:47.212145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:47.212128    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b" event={"ID":"dc7a0bba-95b4-4325-8775-a0a73ff1d67a","Type":"ContainerStarted","Data":"f8b30213b2383eb24cd57f5c36f35cf41d38efe10771bd97060fc122460d90da"}
Nov 02 17:43:47.265830 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:47.265697    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b" podStartSLOduration=1.265675967 podStartE2EDuration="1.265675967s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:47.264030571 +0000 UTC m=+843.755093729" watchObservedRunningTime="2026-11-02 17:43:47.265675967 +0000 UTC m=+843.756739170"
Nov 02 17:43:47.661917 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:47.661886    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:48.662851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:48.662798    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:48.663295 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:48.662865    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b" podUID="dc7a0bba-95b4-4325-8775-a0a73ff1d67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 17:43:49.173845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:49.170912    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:49.662227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:49.662189    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:49.662389 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:49.662241    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b" podUID="dc7a0bba-95b4-4325-8775-a0a73ff1d67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 17:43:50.008578 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.008504    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"]
Nov 02 17:43:50.013212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.013192    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"
Nov 02 17:43:50.028578 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.028479    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"]
Nov 02 17:43:50.040026 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.039997    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:43:50.040159 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.040119    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:43:50.113837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.113775    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ngdst\" (UniqueName: \"kubernetes.io/projected/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-kube-api-access-ngdst\") pod \"maas-api-revoke-sub-51e76e64-rzt6w\" (UID: \"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"
Nov 02 17:43:50.113988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.113926    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51e76e64-rzt6w\" (UID: \"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"
Nov 02 17:43:50.113988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.113974    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51e76e64-rzt6w\" (UID: \"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"
Nov 02 17:43:50.215463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.215422    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ngdst\" (UniqueName: \"kubernetes.io/projected/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-kube-api-access-ngdst\") pod \"maas-api-revoke-sub-51e76e64-rzt6w\" (UID: \"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"
Nov 02 17:43:50.215639 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.215505    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51e76e64-rzt6w\" (UID: \"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"
Nov 02 17:43:50.215639 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.215539    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51e76e64-rzt6w\" (UID: \"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"
Nov 02 17:43:50.216369 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.216295    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-51e76e64-rzt6w\" (UID: \"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"
Nov 02 17:43:50.218025 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.218004    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-51e76e64-rzt6w\" (UID: \"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"
Nov 02 17:43:50.241848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.240998    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ngdst\" (UniqueName: \"kubernetes.io/projected/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-kube-api-access-ngdst\") pod \"maas-api-revoke-sub-51e76e64-rzt6w\" (UID: \"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"
Nov 02 17:43:50.375061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.374984    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"
Nov 02 17:43:50.662837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.662501    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:50.662837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.662562    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b" podUID="dc7a0bba-95b4-4325-8775-a0a73ff1d67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 17:43:50.718492 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:43:50.718460    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode86b922b_7bdc_4b34_ba94_cf10ffa14ed2.slice/crio-7a608061f7009d511e2be96bfc1f447fdf653f0996bc206389c5358ce42f532e WatchSource:0}: Error finding container 7a608061f7009d511e2be96bfc1f447fdf653f0996bc206389c5358ce42f532e: Status 404 returned error can't find the container with id 7a608061f7009d511e2be96bfc1f447fdf653f0996bc206389c5358ce42f532e
Nov 02 17:43:50.718588 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:50.718565    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"]
Nov 02 17:43:51.113012 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:51.110852    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:51.273215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:51.273169    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w" event={"ID":"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2","Type":"ContainerStarted","Data":"070f0b4104b9e8bce8c871f6cff15ff7db04d4c6c8bba4b100cb1f7808e4f428"}
Nov 02 17:43:51.273394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:51.273243    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w" event={"ID":"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2","Type":"ContainerStarted","Data":"7a608061f7009d511e2be96bfc1f447fdf653f0996bc206389c5358ce42f532e"}
Nov 02 17:43:51.326719 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:51.325704    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w" podStartSLOduration=2.325683867 podStartE2EDuration="2.325683867s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:51.320679139 +0000 UTC m=+847.811742319" watchObservedRunningTime="2026-11-02 17:43:51.325683867 +0000 UTC m=+847.816747028"
Nov 02 17:43:51.662167 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:51.662132    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:51.662350 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:51.662186    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b" podUID="dc7a0bba-95b4-4325-8775-a0a73ff1d67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 17:43:52.304282 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:52.304247    4456 generic.go:356] "Generic (PLEG): container finished" podID="e86b922b-7bdc-4b34-ba94-cf10ffa14ed2" containerID="070f0b4104b9e8bce8c871f6cff15ff7db04d4c6c8bba4b100cb1f7808e4f428" exitCode=0
Nov 02 17:43:52.305219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:52.305183    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w" event={"ID":"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2","Type":"ContainerDied","Data":"070f0b4104b9e8bce8c871f6cff15ff7db04d4c6c8bba4b100cb1f7808e4f428"}
Nov 02 17:43:52.662533 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:52.662451    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:52.662533 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:52.662509    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b" podUID="dc7a0bba-95b4-4325-8775-a0a73ff1d67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 17:43:53.022690 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:53.022644    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:53.482678 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:53.482649    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"
Nov 02 17:43:53.572288 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:53.569418    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-maas-api-openshift-ca\") pod \"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2\" (UID: \"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2\") "
Nov 02 17:43:53.572288 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:53.569477    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-maas-api-xks-ca\") pod \"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2\" (UID: \"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2\") "
Nov 02 17:43:53.572288 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:53.569607    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-kube-api-access-ngdst\" (UniqueName: \"kubernetes.io/projected/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-kube-api-access-ngdst\") pod \"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2\" (UID: \"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2\") "
Nov 02 17:43:53.572288 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:53.570172    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-maas-api-openshift-ca" pod "e86b922b-7bdc-4b34-ba94-cf10ffa14ed2" (UID: "e86b922b-7bdc-4b34-ba94-cf10ffa14ed2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:53.572288 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:53.570347    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:53.573237 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:53.573210    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-maas-api-xks-ca" pod "e86b922b-7bdc-4b34-ba94-cf10ffa14ed2" (UID: "e86b922b-7bdc-4b34-ba94-cf10ffa14ed2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:43:53.573617 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:53.573320    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-kube-api-access-ngdst" pod "e86b922b-7bdc-4b34-ba94-cf10ffa14ed2" (UID: "e86b922b-7bdc-4b34-ba94-cf10ffa14ed2"). InnerVolumeSpecName "kube-api-access-ngdst". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:53.661596 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:53.661559    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused" start-of-body=
Nov 02 17:43:53.661789 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:53.661620    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b" podUID="dc7a0bba-95b4-4325-8775-a0a73ff1d67a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.82:15021/healthz/ready\": dial tcp 10.134.0.82:15021: connect: connection refused"
Nov 02 17:43:53.671294 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:53.671267    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:53.671294 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:53.671294    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ngdst\" (UniqueName: \"kubernetes.io/projected/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2-kube-api-access-ngdst\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:54.324350 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:54.324320    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"
Nov 02 17:43:54.325113 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:54.324315    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w" event={"ID":"e86b922b-7bdc-4b34-ba94-cf10ffa14ed2","Type":"ContainerDied","Data":"7a608061f7009d511e2be96bfc1f447fdf653f0996bc206389c5358ce42f532e"}
Nov 02 17:43:54.325196 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:54.325177    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7a608061f7009d511e2be96bfc1f447fdf653f0996bc206389c5358ce42f532e"
Nov 02 17:43:54.722955 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:54.722905    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:54.725342 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:54.723679    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:54.725342 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:54.724571    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:43:54.910486 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:54.910447    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:55.210148 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.210115    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"]
Nov 02 17:43:55.210848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.210827    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e86b922b-7bdc-4b34-ba94-cf10ffa14ed2" containerName="revoke-keys"
Nov 02 17:43:55.210978 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.210849    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e86b922b-7bdc-4b34-ba94-cf10ffa14ed2" containerName="revoke-keys"
Nov 02 17:43:55.211036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.211002    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e86b922b-7bdc-4b34-ba94-cf10ffa14ed2" containerName="revoke-keys"
Nov 02 17:43:55.214664 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.214645    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"
Nov 02 17:43:55.232641 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.232623    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:43:55.233043 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.233010    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:43:55.240231 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.240203    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"]
Nov 02 17:43:55.308078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.308037    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c36e96c-g2gzw\" (UID: \"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"
Nov 02 17:43:55.308315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.308295    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q6hv8\" (UniqueName: \"kubernetes.io/projected/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-kube-api-access-q6hv8\") pod \"maas-api-revoke-sub-4c36e96c-g2gzw\" (UID: \"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"
Nov 02 17:43:55.308445 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.308431    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c36e96c-g2gzw\" (UID: \"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"
Nov 02 17:43:55.410207 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.410106    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q6hv8\" (UniqueName: \"kubernetes.io/projected/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-kube-api-access-q6hv8\") pod \"maas-api-revoke-sub-4c36e96c-g2gzw\" (UID: \"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"
Nov 02 17:43:55.410403 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.410250    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c36e96c-g2gzw\" (UID: \"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"
Nov 02 17:43:55.410403 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.410340    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c36e96c-g2gzw\" (UID: \"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"
Nov 02 17:43:55.411226 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.411198    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4c36e96c-g2gzw\" (UID: \"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"
Nov 02 17:43:55.412938 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.412914    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4c36e96c-g2gzw\" (UID: \"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"
Nov 02 17:43:55.435330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.435303    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q6hv8\" (UniqueName: \"kubernetes.io/projected/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-kube-api-access-q6hv8\") pod \"maas-api-revoke-sub-4c36e96c-g2gzw\" (UID: \"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"
Nov 02 17:43:55.530255 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.530173    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"
Nov 02 17:43:55.846255 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:43:55.846226    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2ac958e4_e1dc_45a6_94e5_d9f2b95e0a99.slice/crio-b6a57fb06880977e7f95c35116622ded65703208eca1ba5530f676aacbed9196 WatchSource:0}: Error finding container b6a57fb06880977e7f95c35116622ded65703208eca1ba5530f676aacbed9196: Status 404 returned error can't find the container with id b6a57fb06880977e7f95c35116622ded65703208eca1ba5530f676aacbed9196
Nov 02 17:43:55.846595 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:55.846444    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"]
Nov 02 17:43:56.340577 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:56.340525    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw" event={"ID":"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99","Type":"ContainerStarted","Data":"117f161c65c576cb2dac9940488bef66b32174a7a3539a091f07e77025705b27"}
Nov 02 17:43:56.340759 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:56.340637    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw" event={"ID":"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99","Type":"ContainerStarted","Data":"b6a57fb06880977e7f95c35116622ded65703208eca1ba5530f676aacbed9196"}
Nov 02 17:43:56.386034 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:56.385928    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw" podStartSLOduration=1.385914633 podStartE2EDuration="1.385914633s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:43:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:43:56.381689207 +0000 UTC m=+852.872752368" watchObservedRunningTime="2026-11-02 17:43:56.385914633 +0000 UTC m=+852.876977804"
Nov 02 17:43:56.386671 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:56.386644    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:56.810047 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:56.809999    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:57.351061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:57.351025    4456 generic.go:356] "Generic (PLEG): container finished" podID="2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99" containerID="117f161c65c576cb2dac9940488bef66b32174a7a3539a091f07e77025705b27" exitCode=0
Nov 02 17:43:57.351444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:57.351081    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw" event={"ID":"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99","Type":"ContainerDied","Data":"117f161c65c576cb2dac9940488bef66b32174a7a3539a091f07e77025705b27"}
Nov 02 17:43:58.564118 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.564088    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"
Nov 02 17:43:58.657691 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.657594    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-maas-api-openshift-ca\") pod \"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99\" (UID: \"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99\") "
Nov 02 17:43:58.657885 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.657687    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-maas-api-xks-ca\") pod \"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99\" (UID: \"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99\") "
Nov 02 17:43:58.657885 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.657749    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-kube-api-access-q6hv8\" (UniqueName: \"kubernetes.io/projected/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-kube-api-access-q6hv8\") pod \"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99\" (UID: \"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99\") "
Nov 02 17:43:58.658097 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.658067    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-maas-api-openshift-ca" pod "2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99" (UID: "2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:43:58.658617 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.658532    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:58.660391 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.660360    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-kube-api-access-q6hv8" pod "2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99" (UID: "2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99"). InnerVolumeSpecName "kube-api-access-q6hv8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:43:58.661155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.661119    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-maas-api-xks-ca" pod "2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99" (UID: "2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:43:58.703905 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.703865    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:43:58.705190 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.705156    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"]
Nov 02 17:43:58.706250 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.706226    4456 cpu_manager.go:414] "containerMap: removing container" podUID="2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99" containerName="revoke-keys"
Nov 02 17:43:58.706250 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.706249    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99" containerName="revoke-keys"
Nov 02 17:43:58.706438 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.706380    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99" containerName="revoke-keys"
Nov 02 17:43:58.710465 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.710445    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"
Nov 02 17:43:58.763888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.763841    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:58.763888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.763876    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q6hv8\" (UniqueName: \"kubernetes.io/projected/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99-kube-api-access-q6hv8\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:43:58.770424 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.770391    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"]
Nov 02 17:43:58.865059 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.865021    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fhqvd\" (UniqueName: \"kubernetes.io/projected/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-kube-api-access-fhqvd\") pod \"maas-api-revoke-sub-b45b8eb8-wrslt\" (UID: \"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"
Nov 02 17:43:58.865253 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.865227    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b45b8eb8-wrslt\" (UID: \"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"
Nov 02 17:43:58.865439 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.865425    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b45b8eb8-wrslt\" (UID: \"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"
Nov 02 17:43:58.966660 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.966621    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b45b8eb8-wrslt\" (UID: \"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"
Nov 02 17:43:58.966883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.966681    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fhqvd\" (UniqueName: \"kubernetes.io/projected/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-kube-api-access-fhqvd\") pod \"maas-api-revoke-sub-b45b8eb8-wrslt\" (UID: \"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"
Nov 02 17:43:58.966883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.966796    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b45b8eb8-wrslt\" (UID: \"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"
Nov 02 17:43:58.967388 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.967363    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b45b8eb8-wrslt\" (UID: \"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"
Nov 02 17:43:58.969307 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.969287    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b45b8eb8-wrslt\" (UID: \"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"
Nov 02 17:43:58.991056 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:58.991027    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fhqvd\" (UniqueName: \"kubernetes.io/projected/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-kube-api-access-fhqvd\") pod \"maas-api-revoke-sub-b45b8eb8-wrslt\" (UID: \"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"
Nov 02 17:43:59.070580 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:59.070544    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"
Nov 02 17:43:59.368028 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:59.367951    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"
Nov 02 17:43:59.368028 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:59.367981    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw" event={"ID":"2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99","Type":"ContainerDied","Data":"b6a57fb06880977e7f95c35116622ded65703208eca1ba5530f676aacbed9196"}
Nov 02 17:43:59.368028 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:59.368028    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b6a57fb06880977e7f95c35116622ded65703208eca1ba5530f676aacbed9196"
Nov 02 17:43:59.409789 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:43:59.409758    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"]
Nov 02 17:43:59.410534 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:43:59.410500    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8cbd5984_ab35_4d6f_8aca_b71af0b88ae9.slice/crio-d5ae3a06a35e204d8266d3987a7697e63b99b38092072fa4ccce6c897ade9d80 WatchSource:0}: Error finding container d5ae3a06a35e204d8266d3987a7697e63b99b38092072fa4ccce6c897ade9d80: Status 404 returned error can't find the container with id d5ae3a06a35e204d8266d3987a7697e63b99b38092072fa4ccce6c897ade9d80
Nov 02 17:44:00.375995 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:00.375965    4456 generic.go:356] "Generic (PLEG): container finished" podID="8cbd5984-ab35-4d6f-8aca-b71af0b88ae9" containerID="403652b698825ca949f4d3dab466777f1fa43423b1043ee7926794fe55ea820c" exitCode=0
Nov 02 17:44:00.376344 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:00.376050    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt" event={"ID":"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9","Type":"ContainerDied","Data":"403652b698825ca949f4d3dab466777f1fa43423b1043ee7926794fe55ea820c"}
Nov 02 17:44:00.376344 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:00.376105    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt" event={"ID":"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9","Type":"ContainerStarted","Data":"d5ae3a06a35e204d8266d3987a7697e63b99b38092072fa4ccce6c897ade9d80"}
Nov 02 17:44:00.620503 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:00.620396    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:01.376313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.376271    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"]
Nov 02 17:44:01.382853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.380881    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.413363 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.413333    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"]
Nov 02 17:44:01.417868 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.417798    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.450873 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.450251    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"]
Nov 02 17:44:01.455848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.454649    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"]
Nov 02 17:44:01.462848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.460076    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.469940 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.468610    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"]
Nov 02 17:44:01.484260 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.484233    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"]
Nov 02 17:44:01.506789 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.506755    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.506789 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.506791    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.507042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.506841    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.507042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.506881    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-x26kd\" (UniqueName: \"kubernetes.io/projected/363eabf7-2a32-4f2c-9333-d03592a4b41d-kube-api-access-x26kd\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.507042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.506900    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.507042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.507000    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388855a1-d9bc-4803-8230-a981530d412d-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.507042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.507024    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.507042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.507038    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.507371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.507061    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.507371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.507083    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388855a1-d9bc-4803-8230-a981530d412d-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.507371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.507104    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.507371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.507118    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgj72\" (UniqueName: \"kubernetes.io/projected/388855a1-d9bc-4803-8230-a981530d412d-kube-api-access-jgj72\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.507371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.507171    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.507371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.507191    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388855a1-d9bc-4803-8230-a981530d412d-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.507371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.507218    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.507371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.507255    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.507371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.507278    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.507371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.507307    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363eabf7-2a32-4f2c-9333-d03592a4b41d-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.608361 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.608328    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.608613 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.608593    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.608781 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.608766    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.608950 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.608933    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-x26kd\" (UniqueName: \"kubernetes.io/projected/363eabf7-2a32-4f2c-9333-d03592a4b41d-kube-api-access-x26kd\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.609062 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.609048    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.609210 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.609195    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.609326 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.609309    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.609463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.609448    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.609596 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.609582    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388855a1-d9bc-4803-8230-a981530d412d-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.609708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.609694    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.609798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.609786    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.609942 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.609927    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.610063 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.609053    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-credential-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.610124 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610043    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388855a1-d9bc-4803-8230-a981530d412d-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.610182 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610126    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.610182 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610159    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgj72\" (UniqueName: \"kubernetes.io/projected/388855a1-d9bc-4803-8230-a981530d412d-kube-api-access-jgj72\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.610296 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610234    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nq9jv\" (UniqueName: \"kubernetes.io/projected/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-kube-api-access-nq9jv\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.610296 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610272    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.610402 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610308    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388855a1-d9bc-4803-8230-a981530d412d-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.610402 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610351    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.610508 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610399    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.610508 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610437    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.610508 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610471    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.610508 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610500    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.610709 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610532    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.610709 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610569    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.610709 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610602    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363eabf7-2a32-4f2c-9333-d03592a4b41d-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.610709 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610634    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.611013 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.610990    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-istio-data\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.611460 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.611439    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-data\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.618147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.611587    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-workload-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.618147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.612139    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-workload-certs\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.618147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.612369    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-workload-socket\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.618147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.608784    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-workload-certs\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.618147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.609063    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-credential-socket\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.618147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.612898    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363eabf7-2a32-4f2c-9333-d03592a4b41d-istiod-ca-cert\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.618147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.613325    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-envoy\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.618147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.613530    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388855a1-d9bc-4803-8230-a981530d412d-istiod-ca-cert\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.618147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.616533    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-istio-envoy\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.618147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.616669    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388855a1-d9bc-4803-8230-a981530d412d-istio-podinfo\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.618147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.617545    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-podinfo\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.635711 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.635646    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-x26kd\" (UniqueName: \"kubernetes.io/projected/363eabf7-2a32-4f2c-9333-d03592a4b41d-kube-api-access-x26kd\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.636858 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.636833    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgj72\" (UniqueName: \"kubernetes.io/projected/388855a1-d9bc-4803-8230-a981530d412d-kube-api-access-jgj72\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.636965 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.636874    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-token\") pod \"e2e-worker-w1-120394-openshift-default-6459698f85-k6slc\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") " pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.641220 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.641200    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388855a1-d9bc-4803-8230-a981530d412d-istio-token\") pod \"e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") " pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.665446 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.665422    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"
Nov 02 17:44:01.712398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.712247    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.712398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.712298    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.712398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.712351    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.713088 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.712807    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nq9jv\" (UniqueName: \"kubernetes.io/projected/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-kube-api-access-nq9jv\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.713192 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.713172    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.713258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.713214    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.713258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.713243    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.713520 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.713281    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.713520 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.713316    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.713643 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.713538    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-workload-certs\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.714592 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.713904    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-workload-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.714592 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.713968    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-data\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.714592 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.714021    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-credential-socket\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.714592 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.714550    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istiod-ca-cert\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.715948 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.715761    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-podinfo\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.716910 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.715973    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:01.716910 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.716427    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-envoy\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.738554 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.738532    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-token\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.741001 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.740973    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nq9jv\" (UniqueName: \"kubernetes.io/projected/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-kube-api-access-nq9jv\") pod \"e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") " pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.766503 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.766464    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:01.792971 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.790151    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:01.821865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.820522    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-maas-api-xks-ca\") pod \"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9\" (UID: \"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9\") "
Nov 02 17:44:01.822634 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.822205    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-maas-api-openshift-ca\") pod \"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9\" (UID: \"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9\") "
Nov 02 17:44:01.822634 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.822574    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-maas-api-openshift-ca" pod "8cbd5984-ab35-4d6f-8aca-b71af0b88ae9" (UID: "8cbd5984-ab35-4d6f-8aca-b71af0b88ae9"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:01.834451 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.827485    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-kube-api-access-fhqvd\" (UniqueName: \"kubernetes.io/projected/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-kube-api-access-fhqvd\") pod \"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9\" (UID: \"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9\") "
Nov 02 17:44:01.834451 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.829025    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:01.834451 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.834379    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-kube-api-access-fhqvd" pod "8cbd5984-ab35-4d6f-8aca-b71af0b88ae9" (UID: "8cbd5984-ab35-4d6f-8aca-b71af0b88ae9"). InnerVolumeSpecName "kube-api-access-fhqvd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:01.837995 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.837950    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-maas-api-xks-ca" pod "8cbd5984-ab35-4d6f-8aca-b71af0b88ae9" (UID: "8cbd5984-ab35-4d6f-8aca-b71af0b88ae9"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:44:01.932102 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.932036    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fhqvd\" (UniqueName: \"kubernetes.io/projected/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-kube-api-access-fhqvd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:01.932102 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:01.932074    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:02.073741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.073708    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"]
Nov 02 17:44:02.073941 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:44:02.073919    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod388855a1_d9bc_4803_8230_a981530d412d.slice/crio-533c3a02fda013e09d6b65662f40e33cef6351e093d783c0a32560279ca1d640 WatchSource:0}: Error finding container 533c3a02fda013e09d6b65662f40e33cef6351e093d783c0a32560279ca1d640: Status 404 returned error can't find the container with id 533c3a02fda013e09d6b65662f40e33cef6351e093d783c0a32560279ca1d640
Nov 02 17:44:02.077025 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.076990    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:02.077120 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.077070    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:02.077120 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.077114    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:02.141211 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.141152    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"]
Nov 02 17:44:02.142123 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:44:02.142098    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod363eabf7_2a32_4f2c_9333_d03592a4b41d.slice/crio-b0cd430826ce1adcaf0c9a7caa94e2e6db5be1fe53b8bdf66863535190c443e9 WatchSource:0}: Error finding container b0cd430826ce1adcaf0c9a7caa94e2e6db5be1fe53b8bdf66863535190c443e9: Status 404 returned error can't find the container with id b0cd430826ce1adcaf0c9a7caa94e2e6db5be1fe53b8bdf66863535190c443e9
Nov 02 17:44:02.144740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.144706    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:02.144865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.144786    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:02.144946 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.144874    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:02.157222 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.157151    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"]
Nov 02 17:44:02.160385 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:44:02.160359    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbabcae9b_991f_4b7b_8d1c_7a4d7804bc24.slice/crio-45cf2750410bf31298fd5b6917007f6227a74f3f418d47ac51390880d71a2a83 WatchSource:0}: Error finding container 45cf2750410bf31298fd5b6917007f6227a74f3f418d47ac51390880d71a2a83: Status 404 returned error can't find the container with id 45cf2750410bf31298fd5b6917007f6227a74f3f418d47ac51390880d71a2a83
Nov 02 17:44:02.163326 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.163296    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:02.163428 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.163358    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:02.163428 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.163390    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:02.426371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.426195    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn" event={"ID":"388855a1-d9bc-4803-8230-a981530d412d","Type":"ContainerStarted","Data":"00866cb105e953f04cf8af424a47330909ec148a18a700abad12199e3f06b775"}
Nov 02 17:44:02.426371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.426268    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn" event={"ID":"388855a1-d9bc-4803-8230-a981530d412d","Type":"ContainerStarted","Data":"533c3a02fda013e09d6b65662f40e33cef6351e093d783c0a32560279ca1d640"}
Nov 02 17:44:02.430155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.428020    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"
Nov 02 17:44:02.430155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.428029    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt" event={"ID":"8cbd5984-ab35-4d6f-8aca-b71af0b88ae9","Type":"ContainerDied","Data":"d5ae3a06a35e204d8266d3987a7697e63b99b38092072fa4ccce6c897ade9d80"}
Nov 02 17:44:02.430155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.428074    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5ae3a06a35e204d8266d3987a7697e63b99b38092072fa4ccce6c897ade9d80"
Nov 02 17:44:02.433884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.433187    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf" event={"ID":"babcae9b-991f-4b7b-8d1c-7a4d7804bc24","Type":"ContainerStarted","Data":"eab8c74e617923c2b748a1867f5951dac5437af3dd2db8e5ecabf5afa94bb225"}
Nov 02 17:44:02.433884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.433234    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf" event={"ID":"babcae9b-991f-4b7b-8d1c-7a4d7804bc24","Type":"ContainerStarted","Data":"45cf2750410bf31298fd5b6917007f6227a74f3f418d47ac51390880d71a2a83"}
Nov 02 17:44:02.435080 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.435052    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc" event={"ID":"363eabf7-2a32-4f2c-9333-d03592a4b41d","Type":"ContainerStarted","Data":"0bcd336e8308ef139181ab90e2f3233572e19d0a9c8c55bb55e13405958b21ef"}
Nov 02 17:44:02.435212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.435103    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc" event={"ID":"363eabf7-2a32-4f2c-9333-d03592a4b41d","Type":"ContainerStarted","Data":"b0cd430826ce1adcaf0c9a7caa94e2e6db5be1fe53b8bdf66863535190c443e9"}
Nov 02 17:44:02.474374 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.474343    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"]
Nov 02 17:44:02.475355 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.475329    4456 cpu_manager.go:414] "containerMap: removing container" podUID="8cbd5984-ab35-4d6f-8aca-b71af0b88ae9" containerName="revoke-keys"
Nov 02 17:44:02.475355 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.475355    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cbd5984-ab35-4d6f-8aca-b71af0b88ae9" containerName="revoke-keys"
Nov 02 17:44:02.475562 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.475546    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cbd5984-ab35-4d6f-8aca-b71af0b88ae9" containerName="revoke-keys"
Nov 02 17:44:02.482853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.480851    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"
Nov 02 17:44:02.485985 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.485935    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn" podStartSLOduration=1.485918534 podStartE2EDuration="1.485918534s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:02.479062137 +0000 UTC m=+858.970125322" watchObservedRunningTime="2026-11-02 17:44:02.485918534 +0000 UTC m=+858.976981695"
Nov 02 17:44:02.566663 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.535157    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:44:02.566663 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.535226    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:44:02.566663 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.541796    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"]
Nov 02 17:44:02.651112 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.645622    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc" podStartSLOduration=1.645601336 podStartE2EDuration="1.645601336s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:02.638961749 +0000 UTC m=+859.130024926" watchObservedRunningTime="2026-11-02 17:44:02.645601336 +0000 UTC m=+859.136664498"
Nov 02 17:44:02.651112 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.647441    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-chcvf\" (UniqueName: \"kubernetes.io/projected/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-kube-api-access-chcvf\") pod \"maas-api-revoke-sub-4ef68676-gm2xp\" (UID: \"1fa2ec57-7deb-424a-ad6b-71dd8caf324c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"
Nov 02 17:44:02.651112 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.647485    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef68676-gm2xp\" (UID: \"1fa2ec57-7deb-424a-ad6b-71dd8caf324c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"
Nov 02 17:44:02.651112 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.647574    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef68676-gm2xp\" (UID: \"1fa2ec57-7deb-424a-ad6b-71dd8caf324c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"
Nov 02 17:44:02.722843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.719921    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:02.757958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.756593    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef68676-gm2xp\" (UID: \"1fa2ec57-7deb-424a-ad6b-71dd8caf324c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"
Nov 02 17:44:02.757958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.757389    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-chcvf\" (UniqueName: \"kubernetes.io/projected/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-kube-api-access-chcvf\") pod \"maas-api-revoke-sub-4ef68676-gm2xp\" (UID: \"1fa2ec57-7deb-424a-ad6b-71dd8caf324c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"
Nov 02 17:44:02.757958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.757421    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef68676-gm2xp\" (UID: \"1fa2ec57-7deb-424a-ad6b-71dd8caf324c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"
Nov 02 17:44:02.765239 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.763047    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf" podStartSLOduration=1.763027516 podStartE2EDuration="1.763027516s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:02.730289703 +0000 UTC m=+859.221352878" watchObservedRunningTime="2026-11-02 17:44:02.763027516 +0000 UTC m=+859.254090678"
Nov 02 17:44:02.766871 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.766843    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:02.768586 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.768540    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4ef68676-gm2xp\" (UID: \"1fa2ec57-7deb-424a-ad6b-71dd8caf324c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"
Nov 02 17:44:02.778834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.776481    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4ef68676-gm2xp\" (UID: \"1fa2ec57-7deb-424a-ad6b-71dd8caf324c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"
Nov 02 17:44:02.791167 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.791140    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:02.797539 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.797500    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:02.824188 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.824015    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-chcvf\" (UniqueName: \"kubernetes.io/projected/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-kube-api-access-chcvf\") pod \"maas-api-revoke-sub-4ef68676-gm2xp\" (UID: \"1fa2ec57-7deb-424a-ad6b-71dd8caf324c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"
Nov 02 17:44:02.905146 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:02.905102    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"
Nov 02 17:44:03.347283 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:03.346910    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"]
Nov 02 17:44:03.506781 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:03.506740    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp" event={"ID":"1fa2ec57-7deb-424a-ad6b-71dd8caf324c","Type":"ContainerStarted","Data":"932c9ecd6837b545ff3d7dd1565f29f6c228ccce855b0587d7405c2db69f66e9"}
Nov 02 17:44:03.802934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:03.796422    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:03.802934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:03.796491    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc" podUID="363eabf7-2a32-4f2c-9333-d03592a4b41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 17:44:03.849135 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:03.849097    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 17:44:03.867840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:03.852876    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn" podUID="388855a1-d9bc-4803-8230-a981530d412d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 17:44:03.869835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:03.869785    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 17:44:03.870270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:03.870131    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf" podUID="babcae9b-991f-4b7b-8d1c-7a4d7804bc24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 17:44:04.556233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:04.556159    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp" event={"ID":"1fa2ec57-7deb-424a-ad6b-71dd8caf324c","Type":"ContainerStarted","Data":"7866b10813614515a69222809e0dbe89b35284b1b018a1020c172d1e3f131fef"}
Nov 02 17:44:04.686378 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:04.686311    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp" podStartSLOduration=2.68629133 podStartE2EDuration="2.68629133s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:04.6841393 +0000 UTC m=+861.175202515" watchObservedRunningTime="2026-11-02 17:44:04.68629133 +0000 UTC m=+861.177354490"
Nov 02 17:44:04.740170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:04.739205    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 17:44:04.740170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:04.739265    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn" podUID="388855a1-d9bc-4803-8230-a981530d412d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 17:44:04.791670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:04.789465    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:04.791670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:04.789524    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc" podUID="363eabf7-2a32-4f2c-9333-d03592a4b41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 17:44:04.791670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:04.791553    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 17:44:04.791670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:04.791594    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf" podUID="babcae9b-991f-4b7b-8d1c-7a4d7804bc24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 17:44:05.370840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:05.358917    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:05.763885 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:05.763428    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 17:44:05.763885 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:05.763488    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn" podUID="388855a1-d9bc-4803-8230-a981530d412d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 17:44:05.777890 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:05.769905    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:05.777890 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:05.769960    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc" podUID="363eabf7-2a32-4f2c-9333-d03592a4b41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 17:44:05.828855 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:05.795658    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 17:44:05.838840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:05.829134    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf" podUID="babcae9b-991f-4b7b-8d1c-7a4d7804bc24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 17:44:06.053051 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:06.051917    4456 generic.go:356] "Generic (PLEG): container finished" podID="1fa2ec57-7deb-424a-ad6b-71dd8caf324c" containerID="7866b10813614515a69222809e0dbe89b35284b1b018a1020c172d1e3f131fef" exitCode=0
Nov 02 17:44:06.053051 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:06.052147    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp" event={"ID":"1fa2ec57-7deb-424a-ad6b-71dd8caf324c","Type":"ContainerDied","Data":"7866b10813614515a69222809e0dbe89b35284b1b018a1020c172d1e3f131fef"}
Nov 02 17:44:06.839545 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:06.838947    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 17:44:06.839545 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:06.839007    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf" podUID="babcae9b-991f-4b7b-8d1c-7a4d7804bc24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 17:44:06.839545 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:06.839429    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:06.839545 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:06.839467    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc" podUID="363eabf7-2a32-4f2c-9333-d03592a4b41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 17:44:07.011843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:06.980502    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 17:44:07.012163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:07.012124    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn" podUID="388855a1-d9bc-4803-8230-a981530d412d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 17:44:07.091893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:07.084915    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:07.177872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:07.164696    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:07.212200 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:07.174912    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:07.594294 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:07.594247    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:07.950839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:07.947238    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:07.950839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:07.947309    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc" podUID="363eabf7-2a32-4f2c-9333-d03592a4b41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 17:44:07.950839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:07.948462    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 17:44:07.950839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:07.948511    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf" podUID="babcae9b-991f-4b7b-8d1c-7a4d7804bc24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 17:44:08.083865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.077674    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:08.085311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.085260    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:08.100227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.087177    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 17:44:08.100509 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.100474    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn" podUID="388855a1-d9bc-4803-8230-a981530d412d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 17:44:08.176854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.161762    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:44:08.421844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.217135    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:08.421844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.379880    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:08.562950 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.241910    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 17:44:08.672496 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.672402    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:44:08.674838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.672975    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:44:08.798839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.700470    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:08.798839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.701202    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:08.798839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.712019    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 02 17:44:08.798839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.787988    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:44:08.877661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.869572    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 17:44:08.877661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.869648    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn" podUID="388855a1-d9bc-4803-8230-a981530d412d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 17:44:08.906260 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.906231    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:08.906511 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.906465    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc" podUID="363eabf7-2a32-4f2c-9333-d03592a4b41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 17:44:08.933981 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.933948    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 17:44:08.934235 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:08.934192    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf" podUID="babcae9b-991f-4b7b-8d1c-7a4d7804bc24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 17:44:09.108841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.098950    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 17:44:09.153860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.079216    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:09.181081 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.180105    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"
Nov 02 17:44:09.200637 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.200603    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:09.202127 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.202096    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:44:09.301141 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.274883    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:44:09.301141 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.275527    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:44:09.304323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.304288    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-maas-api-xks-ca\") pod \"1fa2ec57-7deb-424a-ad6b-71dd8caf324c\" (UID: \"1fa2ec57-7deb-424a-ad6b-71dd8caf324c\") "
Nov 02 17:44:09.305320 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.305295    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-maas-api-openshift-ca\") pod \"1fa2ec57-7deb-424a-ad6b-71dd8caf324c\" (UID: \"1fa2ec57-7deb-424a-ad6b-71dd8caf324c\") "
Nov 02 17:44:09.306143 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.306121    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-kube-api-access-chcvf\" (UniqueName: \"kubernetes.io/projected/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-kube-api-access-chcvf\") pod \"1fa2ec57-7deb-424a-ad6b-71dd8caf324c\" (UID: \"1fa2ec57-7deb-424a-ad6b-71dd8caf324c\") "
Nov 02 17:44:09.359040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.305751    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-maas-api-openshift-ca" pod "1fa2ec57-7deb-424a-ad6b-71dd8caf324c" (UID: "1fa2ec57-7deb-424a-ad6b-71dd8caf324c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:09.359040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.331200    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:44:09.359040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.357844    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:44:09.359040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.357909    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:09.359455 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.359382    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-maas-api-xks-ca" pod "1fa2ec57-7deb-424a-ad6b-71dd8caf324c" (UID: "1fa2ec57-7deb-424a-ad6b-71dd8caf324c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:44:09.378776 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.378729    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:44:09.457289 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.457190    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-kube-api-access-chcvf" pod "1fa2ec57-7deb-424a-ad6b-71dd8caf324c" (UID: "1fa2ec57-7deb-424a-ad6b-71dd8caf324c"). InnerVolumeSpecName "kube-api-access-chcvf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:09.459892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.459803    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-chcvf\" (UniqueName: \"kubernetes.io/projected/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-kube-api-access-chcvf\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:09.459892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.459852    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1fa2ec57-7deb-424a-ad6b-71dd8caf324c-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:09.717638 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.717488    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused" start-of-body=
Nov 02 17:44:09.718005 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.717971    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn" podUID="388855a1-d9bc-4803-8230-a981530d412d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.86:15021/healthz/ready\": dial tcp 10.134.0.86:15021: connect: connection refused"
Nov 02 17:44:09.767950 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.767909    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused" start-of-body=
Nov 02 17:44:09.768167 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.767977    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc" podUID="363eabf7-2a32-4f2c-9333-d03592a4b41d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.87:15021/healthz/ready\": dial tcp 10.134.0.87:15021: connect: connection refused"
Nov 02 17:44:09.791519 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.791480    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused" start-of-body=
Nov 02 17:44:09.791713 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:09.791552    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf" podUID="babcae9b-991f-4b7b-8d1c-7a4d7804bc24" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.88:15021/healthz/ready\": dial tcp 10.134.0.88:15021: connect: connection refused"
Nov 02 17:44:10.242884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:10.235321    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"
Nov 02 17:44:10.242884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:10.236019    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp" event={"ID":"1fa2ec57-7deb-424a-ad6b-71dd8caf324c","Type":"ContainerDied","Data":"932c9ecd6837b545ff3d7dd1565f29f6c228ccce855b0587d7405c2db69f66e9"}
Nov 02 17:44:10.242884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:10.236064    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="932c9ecd6837b545ff3d7dd1565f29f6c228ccce855b0587d7405c2db69f66e9"
Nov 02 17:44:10.773976 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:10.773944    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:10.774677 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:10.774646    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:10.775857 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:10.775803    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:44:10.963914 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:10.959617    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:10.963914 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:10.962072    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:10.963914 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:10.963695    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:44:10.980942 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:10.980906    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:10.981879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:10.981855    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:10.985510 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:10.985486    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:44:12.991874 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:12.980590    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"]
Nov 02 17:44:13.100843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.054651    4456 cpu_manager.go:414] "containerMap: removing container" podUID="1fa2ec57-7deb-424a-ad6b-71dd8caf324c" containerName="revoke-keys"
Nov 02 17:44:13.100843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.054690    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1fa2ec57-7deb-424a-ad6b-71dd8caf324c" containerName="revoke-keys"
Nov 02 17:44:13.100843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.054952    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="1fa2ec57-7deb-424a-ad6b-71dd8caf324c" containerName="revoke-keys"
Nov 02 17:44:13.112840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.112036    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"]
Nov 02 17:44:13.112840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.112125    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:13.112840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.112556    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"
Nov 02 17:44:13.138497 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.119140    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded"
Nov 02 17:44:13.151762 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.145075    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:44:13.151762 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.145145    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:44:13.204940 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.160428    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42081d03-198e-437e-893f-ba984a5b5159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e676391c-khfrt\" (UID: \"42081d03-198e-437e-893f-ba984a5b5159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"
Nov 02 17:44:13.204940 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.160568    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-r2jjn\" (UniqueName: \"kubernetes.io/projected/42081d03-198e-437e-893f-ba984a5b5159-kube-api-access-r2jjn\") pod \"maas-api-revoke-sub-e676391c-khfrt\" (UID: \"42081d03-198e-437e-893f-ba984a5b5159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"
Nov 02 17:44:13.204940 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.160657    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42081d03-198e-437e-893f-ba984a5b5159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e676391c-khfrt\" (UID: \"42081d03-198e-437e-893f-ba984a5b5159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"
Nov 02 17:44:13.318897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.300965    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:44:13.318897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.302974    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42081d03-198e-437e-893f-ba984a5b5159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e676391c-khfrt\" (UID: \"42081d03-198e-437e-893f-ba984a5b5159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"
Nov 02 17:44:13.340851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.332043    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-r2jjn\" (UniqueName: \"kubernetes.io/projected/42081d03-198e-437e-893f-ba984a5b5159-kube-api-access-r2jjn\") pod \"maas-api-revoke-sub-e676391c-khfrt\" (UID: \"42081d03-198e-437e-893f-ba984a5b5159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"
Nov 02 17:44:13.340851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.332234    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42081d03-198e-437e-893f-ba984a5b5159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e676391c-khfrt\" (UID: \"42081d03-198e-437e-893f-ba984a5b5159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"
Nov 02 17:44:13.642929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.480933    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:13.642929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.481366    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:44:13.642929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.526907    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:13.642929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.529859    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:13.642929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.535867    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:13.642929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.536897    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42081d03-198e-437e-893f-ba984a5b5159-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e676391c-khfrt\" (UID: \"42081d03-198e-437e-893f-ba984a5b5159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"
Nov 02 17:44:13.642929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.566943    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:13.642929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.567650    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42081d03-198e-437e-893f-ba984a5b5159-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e676391c-khfrt\" (UID: \"42081d03-198e-437e-893f-ba984a5b5159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"
Nov 02 17:44:13.642929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.569599    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:13.642929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.626649    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-r2jjn\" (UniqueName: \"kubernetes.io/projected/42081d03-198e-437e-893f-ba984a5b5159-kube-api-access-r2jjn\") pod \"maas-api-revoke-sub-e676391c-khfrt\" (UID: \"42081d03-198e-437e-893f-ba984a5b5159\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"
Nov 02 17:44:13.780846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.780258    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"
Nov 02 17:44:13.889841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:13.882901    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:14.110840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.075184    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:14.127969 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.127919    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:14.136843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.128411    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:44:14.330174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.330131    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:14.357686 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.357652    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:44:14.530844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.502873    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:14.530844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.503186    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 17:44:14.530844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.503421    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:44:14.530844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.503476    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:44:14.559151 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.559111    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:14.560457 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.560434    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:44:14.575485 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.575443    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:14.691901 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.602756    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 17:44:14.699837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.695931    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:44:14.699837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.696006    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:44:14.699837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.605702    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:14.788842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.712465    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:44:14.884082 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.884033    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:14.939844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:14.929007    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:44:15.302842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:15.282319    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:15.368676 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:15.368630    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:44:15.752957 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:15.514009    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:15.752957 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:15.514441    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:44:15.822239 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:15.813485    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:15.955881 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:15.955840    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:44:15.957095 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:15.957057    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:15.957284 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:15.957260    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 17:44:16.012904 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:15.987459    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:16.149517 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:15.955911    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 17:44:16.339900 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:16.335973    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:16.377771 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:16.377726    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:16.871833 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:16.871723    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 17:44:16.880343 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:16.876732    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:44:16.910435 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:16.910394    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:16.916420 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:16.916371    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:44:16.916663 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:16.916635    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:16.978843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:16.953101    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:44:16.978843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:16.953176    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.80:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:16.978843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:16.973053    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 17:44:16.981386 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:16.979312    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:44:16.981386 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:16.980235    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:17.164846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:17.155036    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 17:44:17.185209 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:44:17.178924    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:44:17.276844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:17.255919    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:17.336536 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:17.336358    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:44:17.336536 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:17.336429    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr" podUID="fb933398-5485-4600-a38e-640531c54137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:17.363590 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:17.363538    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 17:44:17.406013 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:17.405978    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:44:17.897843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:17.894948    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:17.897843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:17.895335    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 17:44:17.988510 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:17.988372    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:18.030839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:17.998281    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:18.030839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:17.998351    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:44:18.239270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.238987    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"]
Nov 02 17:44:18.283842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.264919    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 17:44:18.283842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.265669    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:18.365967 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.365928    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:18.433989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.433944    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:44:18.446605 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.446078    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.447783 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.447765    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:44:18.448112 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.448090    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt" event={"ID":"42081d03-198e-437e-893f-ba984a5b5159","Type":"ContainerStarted","Data":"cfaed6e90731be7a7ca8a409d0eee4bd7729b802e57aff8015167e507f4ad35d"}
Nov 02 17:44:18.491159 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.491024    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"]
Nov 02 17:44:18.512598 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.512511    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"]
Nov 02 17:44:18.541903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.541054    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.541903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.541103    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.541903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.541132    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.541903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.541228    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.541903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.541318    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dhkgd\" (UniqueName: \"kubernetes.io/projected/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-kube-api-access-dhkgd\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.541903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.541382    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.541903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.541424    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.541903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.541517    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.541903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.541561    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.552512 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.552229    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"]
Nov 02 17:44:18.593344 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.592179    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 02 17:44:18.671179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.644323    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dhkgd\" (UniqueName: \"kubernetes.io/projected/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-kube-api-access-dhkgd\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.671179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.644398    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.671179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.644437    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.671179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.644519    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.671179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.644559    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.671179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.644699    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.671179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.644731    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.671179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.644760    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.671179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.644847    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.671179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.645323    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-workload-certs\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.671179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.645950    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-workload-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.671179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.670462    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istiod-ca-cert\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.692270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.674172    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-credential-socket\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.853911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.847430    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-data\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.853911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.849346    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-token\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.920839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.910919    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:18.920839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.912978    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:18.920839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.913506    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:44:18.964842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.954016    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-envoy\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:18.964842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.954578    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:44:18.989650 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:18.989579    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dhkgd\" (UniqueName: \"kubernetes.io/projected/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-kube-api-access-dhkgd\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:19.072028 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.031936    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:19.072028 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.032229    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 17:44:19.072028 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.032433    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 17:44:19.108898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.104581    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.124842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.121773    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-podinfo\") pod \"e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") " pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:19.202876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.159151    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.202876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.159339    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.202876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.159413    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.202876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.159514    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.202876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.159540    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe469a4c-b038-4869-afb1-5de7dfcebf93-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.202876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.159612    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.202876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.159691    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.202876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.159740    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rwklm\" (UniqueName: \"kubernetes.io/projected/fe469a4c-b038-4869-afb1-5de7dfcebf93-kube-api-access-rwklm\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.217871 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.216239    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:44:19.267241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.259530    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:19.267241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.263519    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.267241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.263577    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rwklm\" (UniqueName: \"kubernetes.io/projected/fe469a4c-b038-4869-afb1-5de7dfcebf93-kube-api-access-rwklm\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.267241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.263682    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.267241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.263801    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.267241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.263931    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.267241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.263977    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.267241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.264043    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.267241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.264068    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe469a4c-b038-4869-afb1-5de7dfcebf93-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.267241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.264119    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.267241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.264565    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.301805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.301394    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.302220 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.302195    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.302639 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.302619    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.303422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.303368    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe469a4c-b038-4869-afb1-5de7dfcebf93-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.303714 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.303698    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.329699 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.310508    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 17:44:19.329699 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.327033    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.353769 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.352841    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rwklm\" (UniqueName: \"kubernetes.io/projected/fe469a4c-b038-4869-afb1-5de7dfcebf93-kube-api-access-rwklm\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.367109 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.364713    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.367109 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.365028    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:44:19.367969 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.367916    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:19.443211 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.428281    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:19.453717 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.453686    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"]
Nov 02 17:44:19.454041 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.454020    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:44:19.476334 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.476307    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:44:19.544613 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.544012    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:44:19.594695 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.594647    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:44:19.596942 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.596772    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:44:19.607691 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.607651    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:19.764791 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.764756    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"]
Nov 02 17:44:19.765472 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.765429    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk" podUID="1a66a57d-f1df-48f1-8715-92a9dab50057" containerName="istio-proxy" containerID="cri-o://b78faf2746e6fa586a94ea96d8e2f14c41ba5043250d3ae9608f82fdb6426bbc" gracePeriod=30
Nov 02 17:44:19.821745 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:19.821685    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt" podStartSLOduration=7.821666757 podStartE2EDuration="7.821666757s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:19.741310506 +0000 UTC m=+876.232373665" watchObservedRunningTime="2026-11-02 17:44:19.821666757 +0000 UTC m=+876.312729919"
Nov 02 17:44:20.018639 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.018605    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"]
Nov 02 17:44:20.021954 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:44:20.021920    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfe469a4c_b038_4869_afb1_5de7dfcebf93.slice/crio-90d1af4366439f70c41d98dd3af9389e9fadaded0a118451f7e8dbc681734725 WatchSource:0}: Error finding container 90d1af4366439f70c41d98dd3af9389e9fadaded0a118451f7e8dbc681734725: Status 404 returned error can't find the container with id 90d1af4366439f70c41d98dd3af9389e9fadaded0a118451f7e8dbc681734725
Nov 02 17:44:20.026128 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.026087    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:20.026233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.026184    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:20.026272 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.026227    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:20.071324 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:44:20.071291    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc5dc9460_ceb7_4c3d_83bc_f038aeb26b8b.slice/crio-d5f95f26ddc6f09846b5a6d0e36d62895ad9306c2317168c3620bf370df8baa9 WatchSource:0}: Error finding container d5f95f26ddc6f09846b5a6d0e36d62895ad9306c2317168c3620bf370df8baa9: Status 404 returned error can't find the container with id d5f95f26ddc6f09846b5a6d0e36d62895ad9306c2317168c3620bf370df8baa9
Nov 02 17:44:20.071996 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.071973    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"]
Nov 02 17:44:20.076371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.076335    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:20.076498 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.076424    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:20.076498 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.076470    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:20.352415 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.351801    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"]
Nov 02 17:44:20.357651 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.357621    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"
Nov 02 17:44:20.372878 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.372094    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"]
Nov 02 17:44:20.451844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.451336    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ghlnh\" (UniqueName: \"kubernetes.io/projected/2a83be4c-c2f5-4392-9a73-ad30b375ae59-kube-api-access-ghlnh\") pod \"maas-api-revoke-sub-5ef19a7c-rtzwp\" (UID: \"2a83be4c-c2f5-4392-9a73-ad30b375ae59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"
Nov 02 17:44:20.451844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.451400    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a83be4c-c2f5-4392-9a73-ad30b375ae59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef19a7c-rtzwp\" (UID: \"2a83be4c-c2f5-4392-9a73-ad30b375ae59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"
Nov 02 17:44:20.451844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.451497    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a83be4c-c2f5-4392-9a73-ad30b375ae59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef19a7c-rtzwp\" (UID: \"2a83be4c-c2f5-4392-9a73-ad30b375ae59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"
Nov 02 17:44:20.554175 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.554135    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a83be4c-c2f5-4392-9a73-ad30b375ae59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef19a7c-rtzwp\" (UID: \"2a83be4c-c2f5-4392-9a73-ad30b375ae59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"
Nov 02 17:44:20.554175 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.554174    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a83be4c-c2f5-4392-9a73-ad30b375ae59-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5ef19a7c-rtzwp\" (UID: \"2a83be4c-c2f5-4392-9a73-ad30b375ae59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"
Nov 02 17:44:20.554545 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.554515    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ghlnh\" (UniqueName: \"kubernetes.io/projected/2a83be4c-c2f5-4392-9a73-ad30b375ae59-kube-api-access-ghlnh\") pod \"maas-api-revoke-sub-5ef19a7c-rtzwp\" (UID: \"2a83be4c-c2f5-4392-9a73-ad30b375ae59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"
Nov 02 17:44:20.554763 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.554733    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a83be4c-c2f5-4392-9a73-ad30b375ae59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef19a7c-rtzwp\" (UID: \"2a83be4c-c2f5-4392-9a73-ad30b375ae59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"
Nov 02 17:44:20.557250 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.557226    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a83be4c-c2f5-4392-9a73-ad30b375ae59-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5ef19a7c-rtzwp\" (UID: \"2a83be4c-c2f5-4392-9a73-ad30b375ae59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"
Nov 02 17:44:20.581274 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.580724    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ghlnh\" (UniqueName: \"kubernetes.io/projected/2a83be4c-c2f5-4392-9a73-ad30b375ae59-kube-api-access-ghlnh\") pod \"maas-api-revoke-sub-5ef19a7c-rtzwp\" (UID: \"2a83be4c-c2f5-4392-9a73-ad30b375ae59\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"
Nov 02 17:44:20.624277 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.624177    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk" event={"ID":"fe469a4c-b038-4869-afb1-5de7dfcebf93","Type":"ContainerStarted","Data":"0f1972bf9560563c35ab21bab295252d0e042c084612634d9a0d162bd1adb1e6"}
Nov 02 17:44:20.624793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.624767    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk" event={"ID":"fe469a4c-b038-4869-afb1-5de7dfcebf93","Type":"ContainerStarted","Data":"90d1af4366439f70c41d98dd3af9389e9fadaded0a118451f7e8dbc681734725"}
Nov 02 17:44:20.630974 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.630945    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh" event={"ID":"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b","Type":"ContainerStarted","Data":"4a2e28ca1c905b35f067d73a0dede8d3690199a6c0405b32fb8b66f98931f6af"}
Nov 02 17:44:20.631112 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.630997    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh" event={"ID":"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b","Type":"ContainerStarted","Data":"d5f95f26ddc6f09846b5a6d0e36d62895ad9306c2317168c3620bf370df8baa9"}
Nov 02 17:44:20.632660 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.632639    4456 generic.go:356] "Generic (PLEG): container finished" podID="42081d03-198e-437e-893f-ba984a5b5159" containerID="2a09f157cd7e1080d1b4c88b58c6f1976a542f38d8e86cb18cf7c5a525e335ae" exitCode=0
Nov 02 17:44:20.632777 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.632719    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt" event={"ID":"42081d03-198e-437e-893f-ba984a5b5159","Type":"ContainerDied","Data":"2a09f157cd7e1080d1b4c88b58c6f1976a542f38d8e86cb18cf7c5a525e335ae"}
Nov 02 17:44:20.676312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.676262    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk" podStartSLOduration=4.676249339 podStartE2EDuration="4.676249339s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:20.674100896 +0000 UTC m=+877.165164093" watchObservedRunningTime="2026-11-02 17:44:20.676249339 +0000 UTC m=+877.167312499"
Nov 02 17:44:20.678313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.678287    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"
Nov 02 17:44:20.837917 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:20.837861    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh" podStartSLOduration=4.837842973 podStartE2EDuration="4.837842973s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:16 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:20.831497503 +0000 UTC m=+877.322560665" watchObservedRunningTime="2026-11-02 17:44:20.837842973 +0000 UTC m=+877.328906128"
Nov 02 17:44:21.018700 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:44:21.018669    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a83be4c_c2f5_4392_9a73_ad30b375ae59.slice/crio-0bfd7f7ec111ea2435bd3b17f8d0ac3ca8fb4a0a158ea259ddaabf7f4ec38589 WatchSource:0}: Error finding container 0bfd7f7ec111ea2435bd3b17f8d0ac3ca8fb4a0a158ea259ddaabf7f4ec38589: Status 404 returned error can't find the container with id 0bfd7f7ec111ea2435bd3b17f8d0ac3ca8fb4a0a158ea259ddaabf7f4ec38589
Nov 02 17:44:21.018946 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.018925    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"]
Nov 02 17:44:21.289646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.278940    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:21.289646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.279105    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 17:44:21.289646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.279147    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh" podUID="c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 17:44:21.354669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.354626    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"]
Nov 02 17:44:21.365690 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.365666    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"
Nov 02 17:44:21.378709 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.378676    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"]
Nov 02 17:44:21.452525 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.452486    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:21.452718 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.452606    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 17:44:21.452718 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.452647    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk" podUID="fe469a4c-b038-4869-afb1-5de7dfcebf93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 17:44:21.474879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.474837    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxql8\" (UniqueName: \"kubernetes.io/projected/d73ea517-8865-490e-a7bd-24d164f8c1a1-kube-api-access-dxql8\") pod \"maas-api-revoke-sub-f2176fc8-7fkb8\" (UID: \"d73ea517-8865-490e-a7bd-24d164f8c1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"
Nov 02 17:44:21.475038 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.474957    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73ea517-8865-490e-a7bd-24d164f8c1a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2176fc8-7fkb8\" (UID: \"d73ea517-8865-490e-a7bd-24d164f8c1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"
Nov 02 17:44:21.475185 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.475164    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73ea517-8865-490e-a7bd-24d164f8c1a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2176fc8-7fkb8\" (UID: \"d73ea517-8865-490e-a7bd-24d164f8c1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"
Nov 02 17:44:21.576461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.576427    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxql8\" (UniqueName: \"kubernetes.io/projected/d73ea517-8865-490e-a7bd-24d164f8c1a1-kube-api-access-dxql8\") pod \"maas-api-revoke-sub-f2176fc8-7fkb8\" (UID: \"d73ea517-8865-490e-a7bd-24d164f8c1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"
Nov 02 17:44:21.576588 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.576499    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73ea517-8865-490e-a7bd-24d164f8c1a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2176fc8-7fkb8\" (UID: \"d73ea517-8865-490e-a7bd-24d164f8c1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"
Nov 02 17:44:21.576641 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.576621    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73ea517-8865-490e-a7bd-24d164f8c1a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2176fc8-7fkb8\" (UID: \"d73ea517-8865-490e-a7bd-24d164f8c1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"
Nov 02 17:44:21.577257 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.577235    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73ea517-8865-490e-a7bd-24d164f8c1a1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f2176fc8-7fkb8\" (UID: \"d73ea517-8865-490e-a7bd-24d164f8c1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"
Nov 02 17:44:21.579301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.579275    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73ea517-8865-490e-a7bd-24d164f8c1a1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f2176fc8-7fkb8\" (UID: \"d73ea517-8865-490e-a7bd-24d164f8c1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"
Nov 02 17:44:21.626840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.625831    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxql8\" (UniqueName: \"kubernetes.io/projected/d73ea517-8865-490e-a7bd-24d164f8c1a1-kube-api-access-dxql8\") pod \"maas-api-revoke-sub-f2176fc8-7fkb8\" (UID: \"d73ea517-8865-490e-a7bd-24d164f8c1a1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"
Nov 02 17:44:21.666764 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.666730    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp" event={"ID":"2a83be4c-c2f5-4392-9a73-ad30b375ae59","Type":"ContainerStarted","Data":"9c7da9ce7016b8c9c0c375716d458a01cb215583addc9df9c1edd085becf8cd8"}
Nov 02 17:44:21.666851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.666790    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp" event={"ID":"2a83be4c-c2f5-4392-9a73-ad30b375ae59","Type":"ContainerStarted","Data":"0bfd7f7ec111ea2435bd3b17f8d0ac3ca8fb4a0a158ea259ddaabf7f4ec38589"}
Nov 02 17:44:21.712891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.711657    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"
Nov 02 17:44:21.737841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.737597    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp" podStartSLOduration=1.7375762300000002 podStartE2EDuration="1.73757623s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:21.71132819 +0000 UTC m=+878.202391377" watchObservedRunningTime="2026-11-02 17:44:21.73757623 +0000 UTC m=+878.228639391"
Nov 02 17:44:21.916920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:21.912693    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:22.270945 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.270916    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 17:44:22.271196 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.271169    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh" podUID="c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 17:44:22.353905 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.351993    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"
Nov 02 17:44:22.377781 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.377753    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/42081d03-198e-437e-893f-ba984a5b5159-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42081d03-198e-437e-893f-ba984a5b5159-maas-api-openshift-ca\") pod \"42081d03-198e-437e-893f-ba984a5b5159\" (UID: \"42081d03-198e-437e-893f-ba984a5b5159\") "
Nov 02 17:44:22.377969 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.377873    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/42081d03-198e-437e-893f-ba984a5b5159-kube-api-access-r2jjn\" (UniqueName: \"kubernetes.io/projected/42081d03-198e-437e-893f-ba984a5b5159-kube-api-access-r2jjn\") pod \"42081d03-198e-437e-893f-ba984a5b5159\" (UID: \"42081d03-198e-437e-893f-ba984a5b5159\") "
Nov 02 17:44:22.377969 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.377929    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/42081d03-198e-437e-893f-ba984a5b5159-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42081d03-198e-437e-893f-ba984a5b5159-maas-api-xks-ca\") pod \"42081d03-198e-437e-893f-ba984a5b5159\" (UID: \"42081d03-198e-437e-893f-ba984a5b5159\") "
Nov 02 17:44:22.378135 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.378112    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/42081d03-198e-437e-893f-ba984a5b5159-maas-api-openshift-ca" pod "42081d03-198e-437e-893f-ba984a5b5159" (UID: "42081d03-198e-437e-893f-ba984a5b5159"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:22.378431 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.378410    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/42081d03-198e-437e-893f-ba984a5b5159-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:22.433562 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.416994    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/42081d03-198e-437e-893f-ba984a5b5159-kube-api-access-r2jjn" pod "42081d03-198e-437e-893f-ba984a5b5159" (UID: "42081d03-198e-437e-893f-ba984a5b5159"). InnerVolumeSpecName "kube-api-access-r2jjn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:22.450839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.447284    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:22.450839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.448902    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/42081d03-198e-437e-893f-ba984a5b5159-maas-api-xks-ca" pod "42081d03-198e-437e-893f-ba984a5b5159" (UID: "42081d03-198e-437e-893f-ba984a5b5159"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:44:22.450839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.448912    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 17:44:22.450839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.448966    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk" podUID="fe469a4c-b038-4869-afb1-5de7dfcebf93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 17:44:22.479683 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.479655    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-r2jjn\" (UniqueName: \"kubernetes.io/projected/42081d03-198e-437e-893f-ba984a5b5159-kube-api-access-r2jjn\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:22.479683 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.479680    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/42081d03-198e-437e-893f-ba984a5b5159-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:22.576382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.576350    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"]
Nov 02 17:44:22.578061 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:44:22.578025    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd73ea517_8865_490e_a7bd_24d164f8c1a1.slice/crio-4122dcbcb8b046825ee7da6293e5224015fad760366c9c17741e90925eae5f6f WatchSource:0}: Error finding container 4122dcbcb8b046825ee7da6293e5224015fad760366c9c17741e90925eae5f6f: Status 404 returned error can't find the container with id 4122dcbcb8b046825ee7da6293e5224015fad760366c9c17741e90925eae5f6f
Nov 02 17:44:22.675399 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.675369    4456 generic.go:356] "Generic (PLEG): container finished" podID="2a83be4c-c2f5-4392-9a73-ad30b375ae59" containerID="9c7da9ce7016b8c9c0c375716d458a01cb215583addc9df9c1edd085becf8cd8" exitCode=0
Nov 02 17:44:22.675520 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.675478    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp" event={"ID":"2a83be4c-c2f5-4392-9a73-ad30b375ae59","Type":"ContainerDied","Data":"9c7da9ce7016b8c9c0c375716d458a01cb215583addc9df9c1edd085becf8cd8"}
Nov 02 17:44:22.677829 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.677792    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"
Nov 02 17:44:22.677829 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.677803    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt" event={"ID":"42081d03-198e-437e-893f-ba984a5b5159","Type":"ContainerDied","Data":"cfaed6e90731be7a7ca8a409d0eee4bd7729b802e57aff8015167e507f4ad35d"}
Nov 02 17:44:22.677972 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.677865    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cfaed6e90731be7a7ca8a409d0eee4bd7729b802e57aff8015167e507f4ad35d"
Nov 02 17:44:22.679387 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:22.679338    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8" event={"ID":"d73ea517-8865-490e-a7bd-24d164f8c1a1","Type":"ContainerStarted","Data":"4122dcbcb8b046825ee7da6293e5224015fad760366c9c17741e90925eae5f6f"}
Nov 02 17:44:23.269783 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:23.269754    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 17:44:23.270000 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:23.269806    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh" podUID="c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 17:44:23.446169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:23.446060    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 17:44:23.446169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:23.446129    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk" podUID="fe469a4c-b038-4869-afb1-5de7dfcebf93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 17:44:23.753465 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:23.752896    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8" event={"ID":"d73ea517-8865-490e-a7bd-24d164f8c1a1","Type":"ContainerStarted","Data":"0f252d8629ebbd1e827fcd456da140add4ec2e64bce2994766ebd2d7897af9d6"}
Nov 02 17:44:23.827842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:23.812627    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8" podStartSLOduration=2.812608953 podStartE2EDuration="2.812608953s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:23.808239011 +0000 UTC m=+880.299302175" watchObservedRunningTime="2026-11-02 17:44:23.812608953 +0000 UTC m=+880.303672115"
Nov 02 17:44:24.114346 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.101041    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:24.269896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.269860    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 17:44:24.270091 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.269909    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh" podUID="c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 17:44:24.455883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.452707    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 17:44:24.455883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.452775    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk" podUID="fe469a4c-b038-4869-afb1-5de7dfcebf93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 17:44:24.460888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.460858    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"
Nov 02 17:44:24.559837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.559028    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a83be4c-c2f5-4392-9a73-ad30b375ae59-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a83be4c-c2f5-4392-9a73-ad30b375ae59-maas-api-openshift-ca\") pod \"2a83be4c-c2f5-4392-9a73-ad30b375ae59\" (UID: \"2a83be4c-c2f5-4392-9a73-ad30b375ae59\") "
Nov 02 17:44:24.559837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.559090    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a83be4c-c2f5-4392-9a73-ad30b375ae59-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a83be4c-c2f5-4392-9a73-ad30b375ae59-maas-api-xks-ca\") pod \"2a83be4c-c2f5-4392-9a73-ad30b375ae59\" (UID: \"2a83be4c-c2f5-4392-9a73-ad30b375ae59\") "
Nov 02 17:44:24.559837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.559135    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a83be4c-c2f5-4392-9a73-ad30b375ae59-kube-api-access-ghlnh\" (UniqueName: \"kubernetes.io/projected/2a83be4c-c2f5-4392-9a73-ad30b375ae59-kube-api-access-ghlnh\") pod \"2a83be4c-c2f5-4392-9a73-ad30b375ae59\" (UID: \"2a83be4c-c2f5-4392-9a73-ad30b375ae59\") "
Nov 02 17:44:24.565114 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.560691    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a83be4c-c2f5-4392-9a73-ad30b375ae59-maas-api-openshift-ca" pod "2a83be4c-c2f5-4392-9a73-ad30b375ae59" (UID: "2a83be4c-c2f5-4392-9a73-ad30b375ae59"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:24.565913 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.565887    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a83be4c-c2f5-4392-9a73-ad30b375ae59-kube-api-access-ghlnh" pod "2a83be4c-c2f5-4392-9a73-ad30b375ae59" (UID: "2a83be4c-c2f5-4392-9a73-ad30b375ae59"). InnerVolumeSpecName "kube-api-access-ghlnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:24.566678 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.566658    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a83be4c-c2f5-4392-9a73-ad30b375ae59-maas-api-xks-ca" pod "2a83be4c-c2f5-4392-9a73-ad30b375ae59" (UID: "2a83be4c-c2f5-4392-9a73-ad30b375ae59"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:44:24.661548 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.661444    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a83be4c-c2f5-4392-9a73-ad30b375ae59-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:24.661548 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.661484    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a83be4c-c2f5-4392-9a73-ad30b375ae59-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:24.661548 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.661499    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ghlnh\" (UniqueName: \"kubernetes.io/projected/2a83be4c-c2f5-4392-9a73-ad30b375ae59-kube-api-access-ghlnh\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:24.762377 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.762340    4456 generic.go:356] "Generic (PLEG): container finished" podID="d73ea517-8865-490e-a7bd-24d164f8c1a1" containerID="0f252d8629ebbd1e827fcd456da140add4ec2e64bce2994766ebd2d7897af9d6" exitCode=0
Nov 02 17:44:24.762539 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.762421    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8" event={"ID":"d73ea517-8865-490e-a7bd-24d164f8c1a1","Type":"ContainerDied","Data":"0f252d8629ebbd1e827fcd456da140add4ec2e64bce2994766ebd2d7897af9d6"}
Nov 02 17:44:24.764170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.764154    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"
Nov 02 17:44:24.764219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.764172    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp" event={"ID":"2a83be4c-c2f5-4392-9a73-ad30b375ae59","Type":"ContainerDied","Data":"0bfd7f7ec111ea2435bd3b17f8d0ac3ca8fb4a0a158ea259ddaabf7f4ec38589"}
Nov 02 17:44:24.764219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:24.764213    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0bfd7f7ec111ea2435bd3b17f8d0ac3ca8fb4a0a158ea259ddaabf7f4ec38589"
Nov 02 17:44:25.269698 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.269611    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 17:44:25.269698 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.269671    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh" podUID="c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 17:44:25.450838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.450178    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 17:44:25.450838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.450236    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk" podUID="fe469a4c-b038-4869-afb1-5de7dfcebf93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 17:44:25.459290 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.459262    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:44:25.503841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.489312    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-podinfo\") pod \"1a66a57d-f1df-48f1-8715-92a9dab50057\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") "
Nov 02 17:44:25.503841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.489496    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-workload-socket\") pod \"1a66a57d-f1df-48f1-8715-92a9dab50057\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") "
Nov 02 17:44:25.503841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.489536    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-data\") pod \"1a66a57d-f1df-48f1-8715-92a9dab50057\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") "
Nov 02 17:44:25.503841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.489664    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a66a57d-f1df-48f1-8715-92a9dab50057-kube-api-access-brj4f\" (UniqueName: \"kubernetes.io/projected/1a66a57d-f1df-48f1-8715-92a9dab50057-kube-api-access-brj4f\") pod \"1a66a57d-f1df-48f1-8715-92a9dab50057\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") "
Nov 02 17:44:25.503841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.489692    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-credential-socket\") pod \"1a66a57d-f1df-48f1-8715-92a9dab50057\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") "
Nov 02 17:44:25.503841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.489782    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-envoy\") pod \"1a66a57d-f1df-48f1-8715-92a9dab50057\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") "
Nov 02 17:44:25.503841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.489847    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1a66a57d-f1df-48f1-8715-92a9dab50057-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a66a57d-f1df-48f1-8715-92a9dab50057-istiod-ca-cert\") pod \"1a66a57d-f1df-48f1-8715-92a9dab50057\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") "
Nov 02 17:44:25.503841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.489881    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-token\" (UniqueName: \"kubernetes.io/projected/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-token\") pod \"1a66a57d-f1df-48f1-8715-92a9dab50057\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") "
Nov 02 17:44:25.503841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.489989    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-workload-certs\") pod \"1a66a57d-f1df-48f1-8715-92a9dab50057\" (UID: \"1a66a57d-f1df-48f1-8715-92a9dab50057\") "
Nov 02 17:44:25.505429 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.505131    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1a66a57d-f1df-48f1-8715-92a9dab50057-istiod-ca-cert" pod "1a66a57d-f1df-48f1-8715-92a9dab50057" (UID: "1a66a57d-f1df-48f1-8715-92a9dab50057"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:25.505928 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.505899    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-workload-socket" pod "1a66a57d-f1df-48f1-8715-92a9dab50057" (UID: "1a66a57d-f1df-48f1-8715-92a9dab50057"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:44:25.528005 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.507915    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-workload-certs" pod "1a66a57d-f1df-48f1-8715-92a9dab50057" (UID: "1a66a57d-f1df-48f1-8715-92a9dab50057"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:44:25.528005 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.515286    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-credential-socket" pod "1a66a57d-f1df-48f1-8715-92a9dab50057" (UID: "1a66a57d-f1df-48f1-8715-92a9dab50057"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:44:25.528005 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.527909    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-data" pod "1a66a57d-f1df-48f1-8715-92a9dab50057" (UID: "1a66a57d-f1df-48f1-8715-92a9dab50057"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:44:25.564295 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.564246    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-podinfo" pod "1a66a57d-f1df-48f1-8715-92a9dab50057" (UID: "1a66a57d-f1df-48f1-8715-92a9dab50057"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:44:25.565024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.564995    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-token" pod "1a66a57d-f1df-48f1-8715-92a9dab50057" (UID: "1a66a57d-f1df-48f1-8715-92a9dab50057"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:25.565173 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.565152    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1a66a57d-f1df-48f1-8715-92a9dab50057-kube-api-access-brj4f" pod "1a66a57d-f1df-48f1-8715-92a9dab50057" (UID: "1a66a57d-f1df-48f1-8715-92a9dab50057"). InnerVolumeSpecName "kube-api-access-brj4f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:25.567909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.567872    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-envoy" pod "1a66a57d-f1df-48f1-8715-92a9dab50057" (UID: "1a66a57d-f1df-48f1-8715-92a9dab50057"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:44:25.607876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.599135    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:25.607876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.599177    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:25.607876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.599191    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:25.607876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.599205    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:25.607876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.599221    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-brj4f\" (UniqueName: \"kubernetes.io/projected/1a66a57d-f1df-48f1-8715-92a9dab50057-kube-api-access-brj4f\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:25.607876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.599235    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:25.607876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.599248    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:25.607876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.599262    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/1a66a57d-f1df-48f1-8715-92a9dab50057-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:25.607876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.599274    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/1a66a57d-f1df-48f1-8715-92a9dab50057-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:25.772695 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.772659    4456 generic.go:356] "Generic (PLEG): container finished" podID="1a66a57d-f1df-48f1-8715-92a9dab50057" containerID="b78faf2746e6fa586a94ea96d8e2f14c41ba5043250d3ae9608f82fdb6426bbc" exitCode=0
Nov 02 17:44:25.772883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.772708    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"
Nov 02 17:44:25.772883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.772727    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk" event={"ID":"1a66a57d-f1df-48f1-8715-92a9dab50057","Type":"ContainerDied","Data":"b78faf2746e6fa586a94ea96d8e2f14c41ba5043250d3ae9608f82fdb6426bbc"}
Nov 02 17:44:25.772883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.772772    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk" event={"ID":"1a66a57d-f1df-48f1-8715-92a9dab50057","Type":"ContainerDied","Data":"b9fd2a4e22caf4eaa8cb6415e5aab7a97b4de54c14abfd143539dd5aa44179b8"}
Nov 02 17:44:25.772883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.772790    4456 scope.go:122] "RemoveContainer" containerID="b78faf2746e6fa586a94ea96d8e2f14c41ba5043250d3ae9608f82fdb6426bbc"
Nov 02 17:44:25.846571 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.846507    4456 scope.go:122] "RemoveContainer" containerID="b78faf2746e6fa586a94ea96d8e2f14c41ba5043250d3ae9608f82fdb6426bbc"
Nov 02 17:44:25.847004 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:44:25.846978    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b78faf2746e6fa586a94ea96d8e2f14c41ba5043250d3ae9608f82fdb6426bbc\": container with ID starting with b78faf2746e6fa586a94ea96d8e2f14c41ba5043250d3ae9608f82fdb6426bbc not found: ID does not exist" containerID="b78faf2746e6fa586a94ea96d8e2f14c41ba5043250d3ae9608f82fdb6426bbc"
Nov 02 17:44:25.847128 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.847016    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b78faf2746e6fa586a94ea96d8e2f14c41ba5043250d3ae9608f82fdb6426bbc"} err="failed to get container status \"b78faf2746e6fa586a94ea96d8e2f14c41ba5043250d3ae9608f82fdb6426bbc\": rpc error: code = NotFound desc = could not find container \"b78faf2746e6fa586a94ea96d8e2f14c41ba5043250d3ae9608f82fdb6426bbc\": container with ID starting with b78faf2746e6fa586a94ea96d8e2f14c41ba5043250d3ae9608f82fdb6426bbc not found: ID does not exist"
Nov 02 17:44:25.873767 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.873727    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"]
Nov 02 17:44:25.879769 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:25.879739    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-c5fdk"]
Nov 02 17:44:26.095677 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.095418    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:26.272386 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.272343    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 17:44:26.272665 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.272634    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh" podUID="c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 17:44:26.329358 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.329330    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"
Nov 02 17:44:26.449948 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.449199    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 17:44:26.449948 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.449268    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk" podUID="fe469a4c-b038-4869-afb1-5de7dfcebf93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 17:44:26.453085 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.453045    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d73ea517-8865-490e-a7bd-24d164f8c1a1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73ea517-8865-490e-a7bd-24d164f8c1a1-maas-api-openshift-ca\") pod \"d73ea517-8865-490e-a7bd-24d164f8c1a1\" (UID: \"d73ea517-8865-490e-a7bd-24d164f8c1a1\") "
Nov 02 17:44:26.453234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.453093    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d73ea517-8865-490e-a7bd-24d164f8c1a1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73ea517-8865-490e-a7bd-24d164f8c1a1-maas-api-xks-ca\") pod \"d73ea517-8865-490e-a7bd-24d164f8c1a1\" (UID: \"d73ea517-8865-490e-a7bd-24d164f8c1a1\") "
Nov 02 17:44:26.453353 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.453339    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d73ea517-8865-490e-a7bd-24d164f8c1a1-kube-api-access-dxql8\" (UniqueName: \"kubernetes.io/projected/d73ea517-8865-490e-a7bd-24d164f8c1a1-kube-api-access-dxql8\") pod \"d73ea517-8865-490e-a7bd-24d164f8c1a1\" (UID: \"d73ea517-8865-490e-a7bd-24d164f8c1a1\") "
Nov 02 17:44:26.454533 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.454510    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d73ea517-8865-490e-a7bd-24d164f8c1a1-maas-api-openshift-ca" pod "d73ea517-8865-490e-a7bd-24d164f8c1a1" (UID: "d73ea517-8865-490e-a7bd-24d164f8c1a1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:26.456878 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.456853    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d73ea517-8865-490e-a7bd-24d164f8c1a1-maas-api-xks-ca" pod "d73ea517-8865-490e-a7bd-24d164f8c1a1" (UID: "d73ea517-8865-490e-a7bd-24d164f8c1a1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:44:26.458167 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.458147    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d73ea517-8865-490e-a7bd-24d164f8c1a1-kube-api-access-dxql8" pod "d73ea517-8865-490e-a7bd-24d164f8c1a1" (UID: "d73ea517-8865-490e-a7bd-24d164f8c1a1"). InnerVolumeSpecName "kube-api-access-dxql8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:26.555652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.555618    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d73ea517-8865-490e-a7bd-24d164f8c1a1-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:26.555652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.555654    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d73ea517-8865-490e-a7bd-24d164f8c1a1-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:26.556150 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.555669    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxql8\" (UniqueName: \"kubernetes.io/projected/d73ea517-8865-490e-a7bd-24d164f8c1a1-kube-api-access-dxql8\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:26.809893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.808884    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8" event={"ID":"d73ea517-8865-490e-a7bd-24d164f8c1a1","Type":"ContainerDied","Data":"4122dcbcb8b046825ee7da6293e5224015fad760366c9c17741e90925eae5f6f"}
Nov 02 17:44:26.809893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.808961    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4122dcbcb8b046825ee7da6293e5224015fad760366c9c17741e90925eae5f6f"
Nov 02 17:44:26.809893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:26.809034    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"
Nov 02 17:44:27.270190 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:27.270145    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused" start-of-body=
Nov 02 17:44:27.270421 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:27.270227    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh" podUID="c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.91:15021/healthz/ready\": dial tcp 10.134.0.91:15021: connect: connection refused"
Nov 02 17:44:27.446505 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:27.446456    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 17:44:27.446706 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:27.446533    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk" podUID="fe469a4c-b038-4869-afb1-5de7dfcebf93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 17:44:27.617501 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:27.617404    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1a66a57d-f1df-48f1-8715-92a9dab50057" path="/var/lib/kubelet/pods/1a66a57d-f1df-48f1-8715-92a9dab50057/volumes"
Nov 02 17:44:28.056838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.047689    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:28.274415 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.274387    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:28.274870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.274848    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:28.275700 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.275678    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:44:28.451338 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.451311    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:28.451865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.451842    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:28.452534 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.452515    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 17:44:28.832569 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.832491    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"]
Nov 02 17:44:28.833438 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.833417    4456 cpu_manager.go:414] "containerMap: removing container" podUID="1a66a57d-f1df-48f1-8715-92a9dab50057" containerName="istio-proxy"
Nov 02 17:44:28.833482 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.833442    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1a66a57d-f1df-48f1-8715-92a9dab50057" containerName="istio-proxy"
Nov 02 17:44:28.833482 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.833466    4456 cpu_manager.go:414] "containerMap: removing container" podUID="2a83be4c-c2f5-4392-9a73-ad30b375ae59" containerName="revoke-keys"
Nov 02 17:44:28.833482 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.833474    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a83be4c-c2f5-4392-9a73-ad30b375ae59" containerName="revoke-keys"
Nov 02 17:44:28.833575 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.833506    4456 cpu_manager.go:414] "containerMap: removing container" podUID="42081d03-198e-437e-893f-ba984a5b5159" containerName="revoke-keys"
Nov 02 17:44:28.833575 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.833514    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="42081d03-198e-437e-893f-ba984a5b5159" containerName="revoke-keys"
Nov 02 17:44:28.833575 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.833572    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d73ea517-8865-490e-a7bd-24d164f8c1a1" containerName="revoke-keys"
Nov 02 17:44:28.833668 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.833581    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d73ea517-8865-490e-a7bd-24d164f8c1a1" containerName="revoke-keys"
Nov 02 17:44:28.833723 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.833712    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d73ea517-8865-490e-a7bd-24d164f8c1a1" containerName="revoke-keys"
Nov 02 17:44:28.833754 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.833730    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="1a66a57d-f1df-48f1-8715-92a9dab50057" containerName="istio-proxy"
Nov 02 17:44:28.833754 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.833741    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a83be4c-c2f5-4392-9a73-ad30b375ae59" containerName="revoke-keys"
Nov 02 17:44:28.833845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.833756    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="42081d03-198e-437e-893f-ba984a5b5159" containerName="revoke-keys"
Nov 02 17:44:28.838534 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.838514    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"
Nov 02 17:44:28.854929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.854891    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:44:28.855094 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.855069    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"]
Nov 02 17:44:28.855283 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.855142    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:44:28.946684 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.946646    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-74gnh\" (UniqueName: \"kubernetes.io/projected/da476c88-7665-46d9-917e-db0100e78636-kube-api-access-74gnh\") pod \"maas-api-revoke-sub-da9459d3-84fdr\" (UID: \"da476c88-7665-46d9-917e-db0100e78636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"
Nov 02 17:44:28.946925 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.946903    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da476c88-7665-46d9-917e-db0100e78636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da9459d3-84fdr\" (UID: \"da476c88-7665-46d9-917e-db0100e78636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"
Nov 02 17:44:28.946996 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:28.946947    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da476c88-7665-46d9-917e-db0100e78636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da9459d3-84fdr\" (UID: \"da476c88-7665-46d9-917e-db0100e78636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"
Nov 02 17:44:29.048870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:29.048829    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da476c88-7665-46d9-917e-db0100e78636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da9459d3-84fdr\" (UID: \"da476c88-7665-46d9-917e-db0100e78636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"
Nov 02 17:44:29.049034 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:29.048886    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da476c88-7665-46d9-917e-db0100e78636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da9459d3-84fdr\" (UID: \"da476c88-7665-46d9-917e-db0100e78636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"
Nov 02 17:44:29.049034 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:29.049014    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-74gnh\" (UniqueName: \"kubernetes.io/projected/da476c88-7665-46d9-917e-db0100e78636-kube-api-access-74gnh\") pod \"maas-api-revoke-sub-da9459d3-84fdr\" (UID: \"da476c88-7665-46d9-917e-db0100e78636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"
Nov 02 17:44:29.049534 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:29.049513    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da476c88-7665-46d9-917e-db0100e78636-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-da9459d3-84fdr\" (UID: \"da476c88-7665-46d9-917e-db0100e78636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"
Nov 02 17:44:29.051206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:29.051182    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da476c88-7665-46d9-917e-db0100e78636-maas-api-xks-ca\") pod \"maas-api-revoke-sub-da9459d3-84fdr\" (UID: \"da476c88-7665-46d9-917e-db0100e78636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"
Nov 02 17:44:29.075306 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:29.075284    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-74gnh\" (UniqueName: \"kubernetes.io/projected/da476c88-7665-46d9-917e-db0100e78636-kube-api-access-74gnh\") pod \"maas-api-revoke-sub-da9459d3-84fdr\" (UID: \"da476c88-7665-46d9-917e-db0100e78636\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"
Nov 02 17:44:29.157474 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:29.157393    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"
Nov 02 17:44:29.485223 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:29.480257    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"]
Nov 02 17:44:29.488289 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:44:29.488253    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podda476c88_7665_46d9_917e_db0100e78636.slice/crio-4172ee91ac7a896e6c90fc02a19b7973df0ef4d245fe14e60873e856a42607c0 WatchSource:0}: Error finding container 4172ee91ac7a896e6c90fc02a19b7973df0ef4d245fe14e60873e856a42607c0: Status 404 returned error can't find the container with id 4172ee91ac7a896e6c90fc02a19b7973df0ef4d245fe14e60873e856a42607c0
Nov 02 17:44:29.841958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:29.841914    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr" event={"ID":"da476c88-7665-46d9-917e-db0100e78636","Type":"ContainerStarted","Data":"7db6504040396951ad899da09d4506da0ce0fe4b8fbcf7676cefb48c88a256a6"}
Nov 02 17:44:29.842353 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:29.841984    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr" event={"ID":"da476c88-7665-46d9-917e-db0100e78636","Type":"ContainerStarted","Data":"4172ee91ac7a896e6c90fc02a19b7973df0ef4d245fe14e60873e856a42607c0"}
Nov 02 17:44:29.959670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:29.959625    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:30.854066 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:30.854032    4456 generic.go:356] "Generic (PLEG): container finished" podID="da476c88-7665-46d9-917e-db0100e78636" containerID="7db6504040396951ad899da09d4506da0ce0fe4b8fbcf7676cefb48c88a256a6" exitCode=0
Nov 02 17:44:30.854532 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:30.854095    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr" event={"ID":"da476c88-7665-46d9-917e-db0100e78636","Type":"ContainerDied","Data":"7db6504040396951ad899da09d4506da0ce0fe4b8fbcf7676cefb48c88a256a6"}
Nov 02 17:44:32.005392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:32.005317    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:32.064748 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:32.064720    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"
Nov 02 17:44:32.106411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:32.106380    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/da476c88-7665-46d9-917e-db0100e78636-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da476c88-7665-46d9-917e-db0100e78636-maas-api-openshift-ca\") pod \"da476c88-7665-46d9-917e-db0100e78636\" (UID: \"da476c88-7665-46d9-917e-db0100e78636\") "
Nov 02 17:44:32.106576 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:32.106516    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/da476c88-7665-46d9-917e-db0100e78636-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da476c88-7665-46d9-917e-db0100e78636-maas-api-xks-ca\") pod \"da476c88-7665-46d9-917e-db0100e78636\" (UID: \"da476c88-7665-46d9-917e-db0100e78636\") "
Nov 02 17:44:32.106712 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:32.106691    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/da476c88-7665-46d9-917e-db0100e78636-kube-api-access-74gnh\" (UniqueName: \"kubernetes.io/projected/da476c88-7665-46d9-917e-db0100e78636-kube-api-access-74gnh\") pod \"da476c88-7665-46d9-917e-db0100e78636\" (UID: \"da476c88-7665-46d9-917e-db0100e78636\") "
Nov 02 17:44:32.106838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:32.106797    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/da476c88-7665-46d9-917e-db0100e78636-maas-api-openshift-ca" pod "da476c88-7665-46d9-917e-db0100e78636" (UID: "da476c88-7665-46d9-917e-db0100e78636"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:32.107314 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:32.107291    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/da476c88-7665-46d9-917e-db0100e78636-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:32.108767 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:32.108749    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/da476c88-7665-46d9-917e-db0100e78636-maas-api-xks-ca" pod "da476c88-7665-46d9-917e-db0100e78636" (UID: "da476c88-7665-46d9-917e-db0100e78636"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:44:32.109007 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:32.108990    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/da476c88-7665-46d9-917e-db0100e78636-kube-api-access-74gnh" pod "da476c88-7665-46d9-917e-db0100e78636" (UID: "da476c88-7665-46d9-917e-db0100e78636"). InnerVolumeSpecName "kube-api-access-74gnh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:32.208365 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:32.208331    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-74gnh\" (UniqueName: \"kubernetes.io/projected/da476c88-7665-46d9-917e-db0100e78636-kube-api-access-74gnh\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:32.208365 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:32.208361    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/da476c88-7665-46d9-917e-db0100e78636-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:32.872935 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:32.872903    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"
Nov 02 17:44:32.873113 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:32.872947    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr" event={"ID":"da476c88-7665-46d9-917e-db0100e78636","Type":"ContainerDied","Data":"4172ee91ac7a896e6c90fc02a19b7973df0ef4d245fe14e60873e856a42607c0"}
Nov 02 17:44:32.873113 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:32.873003    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4172ee91ac7a896e6c90fc02a19b7973df0ef4d245fe14e60873e856a42607c0"
Nov 02 17:44:33.867514 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:33.867473    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:34.049209 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:34.049179    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:35.245240 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:35.245211    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:35.425542 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:35.425500    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:35.957755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:35.944951    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:37.617027 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.616993    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"]
Nov 02 17:44:37.617624 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.617607    4456 cpu_manager.go:414] "containerMap: removing container" podUID="da476c88-7665-46d9-917e-db0100e78636" containerName="revoke-keys"
Nov 02 17:44:37.617624 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.617624    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="da476c88-7665-46d9-917e-db0100e78636" containerName="revoke-keys"
Nov 02 17:44:37.617761 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.617722    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="da476c88-7665-46d9-917e-db0100e78636" containerName="revoke-keys"
Nov 02 17:44:37.622756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.622733    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"
Nov 02 17:44:37.635356 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.635329    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"]
Nov 02 17:44:37.642114 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.642087    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:44:37.642114 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.642095    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:44:37.697997 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.697946    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/372e3319-752f-43bb-8c10-197c19f9a3aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-869d79c6-28cxr\" (UID: \"372e3319-752f-43bb-8c10-197c19f9a3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"
Nov 02 17:44:37.698177 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.698112    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lpfd4\" (UniqueName: \"kubernetes.io/projected/372e3319-752f-43bb-8c10-197c19f9a3aa-kube-api-access-lpfd4\") pod \"maas-api-revoke-sub-869d79c6-28cxr\" (UID: \"372e3319-752f-43bb-8c10-197c19f9a3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"
Nov 02 17:44:37.698495 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.698463    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/372e3319-752f-43bb-8c10-197c19f9a3aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-869d79c6-28cxr\" (UID: \"372e3319-752f-43bb-8c10-197c19f9a3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"
Nov 02 17:44:37.800428 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.800397    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/372e3319-752f-43bb-8c10-197c19f9a3aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-869d79c6-28cxr\" (UID: \"372e3319-752f-43bb-8c10-197c19f9a3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"
Nov 02 17:44:37.800641 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.800480    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/372e3319-752f-43bb-8c10-197c19f9a3aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-869d79c6-28cxr\" (UID: \"372e3319-752f-43bb-8c10-197c19f9a3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"
Nov 02 17:44:37.800641 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.800542    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lpfd4\" (UniqueName: \"kubernetes.io/projected/372e3319-752f-43bb-8c10-197c19f9a3aa-kube-api-access-lpfd4\") pod \"maas-api-revoke-sub-869d79c6-28cxr\" (UID: \"372e3319-752f-43bb-8c10-197c19f9a3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"
Nov 02 17:44:37.801149 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.801119    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/372e3319-752f-43bb-8c10-197c19f9a3aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-869d79c6-28cxr\" (UID: \"372e3319-752f-43bb-8c10-197c19f9a3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"
Nov 02 17:44:37.803223 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.803198    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/372e3319-752f-43bb-8c10-197c19f9a3aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-869d79c6-28cxr\" (UID: \"372e3319-752f-43bb-8c10-197c19f9a3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"
Nov 02 17:44:37.825575 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.825543    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lpfd4\" (UniqueName: \"kubernetes.io/projected/372e3319-752f-43bb-8c10-197c19f9a3aa-kube-api-access-lpfd4\") pod \"maas-api-revoke-sub-869d79c6-28cxr\" (UID: \"372e3319-752f-43bb-8c10-197c19f9a3aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"
Nov 02 17:44:37.871864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.871736    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:37.941539 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:37.941509    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"
Nov 02 17:44:38.275224 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:38.275195    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"]
Nov 02 17:44:38.275832 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:44:38.275782    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod372e3319_752f_43bb_8c10_197c19f9a3aa.slice/crio-4e977a0a0aa22c1c464e634acd58babaf00aa328d85caedd508dea87560c36a3 WatchSource:0}: Error finding container 4e977a0a0aa22c1c464e634acd58babaf00aa328d85caedd508dea87560c36a3: Status 404 returned error can't find the container with id 4e977a0a0aa22c1c464e634acd58babaf00aa328d85caedd508dea87560c36a3
Nov 02 17:44:38.947839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:38.947775    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr" event={"ID":"372e3319-752f-43bb-8c10-197c19f9a3aa","Type":"ContainerStarted","Data":"5781bfe9ba31568fa9879ba01ecac6b559a940287532dad8bfc3e6be3fb46bad"}
Nov 02 17:44:38.948206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:38.947854    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr" event={"ID":"372e3319-752f-43bb-8c10-197c19f9a3aa","Type":"ContainerStarted","Data":"4e977a0a0aa22c1c464e634acd58babaf00aa328d85caedd508dea87560c36a3"}
Nov 02 17:44:39.001463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:39.001412    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr" podStartSLOduration=2.001397539 podStartE2EDuration="2.001397539s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:38.997667974 +0000 UTC m=+895.488731155" watchObservedRunningTime="2026-11-02 17:44:39.001397539 +0000 UTC m=+895.492460747"
Nov 02 17:44:39.782850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:39.782770    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:39.967324 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:39.967292    4456 generic.go:356] "Generic (PLEG): container finished" podID="372e3319-752f-43bb-8c10-197c19f9a3aa" containerID="5781bfe9ba31568fa9879ba01ecac6b559a940287532dad8bfc3e6be3fb46bad" exitCode=0
Nov 02 17:44:39.967715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:39.967512    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr" event={"ID":"372e3319-752f-43bb-8c10-197c19f9a3aa","Type":"ContainerDied","Data":"5781bfe9ba31568fa9879ba01ecac6b559a940287532dad8bfc3e6be3fb46bad"}
Nov 02 17:44:41.166795 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:41.166767    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"
Nov 02 17:44:41.283674 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:41.283572    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/372e3319-752f-43bb-8c10-197c19f9a3aa-kube-api-access-lpfd4\" (UniqueName: \"kubernetes.io/projected/372e3319-752f-43bb-8c10-197c19f9a3aa-kube-api-access-lpfd4\") pod \"372e3319-752f-43bb-8c10-197c19f9a3aa\" (UID: \"372e3319-752f-43bb-8c10-197c19f9a3aa\") "
Nov 02 17:44:41.283981 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:41.283949    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/372e3319-752f-43bb-8c10-197c19f9a3aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/372e3319-752f-43bb-8c10-197c19f9a3aa-maas-api-xks-ca\") pod \"372e3319-752f-43bb-8c10-197c19f9a3aa\" (UID: \"372e3319-752f-43bb-8c10-197c19f9a3aa\") "
Nov 02 17:44:41.284072 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:41.284003    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/372e3319-752f-43bb-8c10-197c19f9a3aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/372e3319-752f-43bb-8c10-197c19f9a3aa-maas-api-openshift-ca\") pod \"372e3319-752f-43bb-8c10-197c19f9a3aa\" (UID: \"372e3319-752f-43bb-8c10-197c19f9a3aa\") "
Nov 02 17:44:41.284681 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:41.284646    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/372e3319-752f-43bb-8c10-197c19f9a3aa-maas-api-openshift-ca" pod "372e3319-752f-43bb-8c10-197c19f9a3aa" (UID: "372e3319-752f-43bb-8c10-197c19f9a3aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:41.286323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:41.286302    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/372e3319-752f-43bb-8c10-197c19f9a3aa-kube-api-access-lpfd4" pod "372e3319-752f-43bb-8c10-197c19f9a3aa" (UID: "372e3319-752f-43bb-8c10-197c19f9a3aa"). InnerVolumeSpecName "kube-api-access-lpfd4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:41.287048 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:41.287022    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/372e3319-752f-43bb-8c10-197c19f9a3aa-maas-api-xks-ca" pod "372e3319-752f-43bb-8c10-197c19f9a3aa" (UID: "372e3319-752f-43bb-8c10-197c19f9a3aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:44:41.385907 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:41.385876    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lpfd4\" (UniqueName: \"kubernetes.io/projected/372e3319-752f-43bb-8c10-197c19f9a3aa-kube-api-access-lpfd4\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:41.385907 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:41.385905    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/372e3319-752f-43bb-8c10-197c19f9a3aa-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:41.385907 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:41.385915    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/372e3319-752f-43bb-8c10-197c19f9a3aa-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:41.937543 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:41.937445    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:41.990070 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:41.990030    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr" event={"ID":"372e3319-752f-43bb-8c10-197c19f9a3aa","Type":"ContainerDied","Data":"4e977a0a0aa22c1c464e634acd58babaf00aa328d85caedd508dea87560c36a3"}
Nov 02 17:44:41.990291 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:41.990103    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4e977a0a0aa22c1c464e634acd58babaf00aa328d85caedd508dea87560c36a3"
Nov 02 17:44:41.990291 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:41.990045    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"
Nov 02 17:44:42.520420 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.520387    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"]
Nov 02 17:44:42.521425 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.521403    4456 cpu_manager.go:414] "containerMap: removing container" podUID="372e3319-752f-43bb-8c10-197c19f9a3aa" containerName="revoke-keys"
Nov 02 17:44:42.521486 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.521430    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="372e3319-752f-43bb-8c10-197c19f9a3aa" containerName="revoke-keys"
Nov 02 17:44:42.521599 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.521587    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="372e3319-752f-43bb-8c10-197c19f9a3aa" containerName="revoke-keys"
Nov 02 17:44:42.526061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.526039    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"
Nov 02 17:44:42.543893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.543864    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"]
Nov 02 17:44:42.549884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.549831    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:44:42.550024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.550003    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:44:42.608934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.608900    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6v9h9\" (UniqueName: \"kubernetes.io/projected/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-kube-api-access-6v9h9\") pod \"maas-api-revoke-sub-827c31a3-hg95f\" (UID: \"50522edc-3a5e-4c72-a147-8b99ba6c9dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"
Nov 02 17:44:42.609117 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.608973    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-827c31a3-hg95f\" (UID: \"50522edc-3a5e-4c72-a147-8b99ba6c9dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"
Nov 02 17:44:42.609248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.609225    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-827c31a3-hg95f\" (UID: \"50522edc-3a5e-4c72-a147-8b99ba6c9dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"
Nov 02 17:44:42.710964 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.710924    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-827c31a3-hg95f\" (UID: \"50522edc-3a5e-4c72-a147-8b99ba6c9dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"
Nov 02 17:44:42.711177 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.711153    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6v9h9\" (UniqueName: \"kubernetes.io/projected/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-kube-api-access-6v9h9\") pod \"maas-api-revoke-sub-827c31a3-hg95f\" (UID: \"50522edc-3a5e-4c72-a147-8b99ba6c9dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"
Nov 02 17:44:42.711264 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.711238    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-827c31a3-hg95f\" (UID: \"50522edc-3a5e-4c72-a147-8b99ba6c9dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"
Nov 02 17:44:42.711716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.711691    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-827c31a3-hg95f\" (UID: \"50522edc-3a5e-4c72-a147-8b99ba6c9dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"
Nov 02 17:44:42.713843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.713800    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-maas-api-xks-ca\") pod \"maas-api-revoke-sub-827c31a3-hg95f\" (UID: \"50522edc-3a5e-4c72-a147-8b99ba6c9dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"
Nov 02 17:44:42.737027 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.736998    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6v9h9\" (UniqueName: \"kubernetes.io/projected/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-kube-api-access-6v9h9\") pod \"maas-api-revoke-sub-827c31a3-hg95f\" (UID: \"50522edc-3a5e-4c72-a147-8b99ba6c9dc1\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"
Nov 02 17:44:42.844683 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:42.844591    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"
Nov 02 17:44:43.160956 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:44:43.160924    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod50522edc_3a5e_4c72_a147_8b99ba6c9dc1.slice/crio-8cbcf7dd63ad35e2b5c304de7d01c4a0fd98f322debfd9d49468599f30c2a02a WatchSource:0}: Error finding container 8cbcf7dd63ad35e2b5c304de7d01c4a0fd98f322debfd9d49468599f30c2a02a: Status 404 returned error can't find the container with id 8cbcf7dd63ad35e2b5c304de7d01c4a0fd98f322debfd9d49468599f30c2a02a
Nov 02 17:44:43.163553 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:43.163526    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"]
Nov 02 17:44:44.008103 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:44.008069    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f" event={"ID":"50522edc-3a5e-4c72-a147-8b99ba6c9dc1","Type":"ContainerStarted","Data":"883eff289f03da617853c7e0d1141e259b3c0a03e1b0569a985e9ed7312a83d2"}
Nov 02 17:44:44.008485 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:44.008123    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f" event={"ID":"50522edc-3a5e-4c72-a147-8b99ba6c9dc1","Type":"ContainerStarted","Data":"8cbcf7dd63ad35e2b5c304de7d01c4a0fd98f322debfd9d49468599f30c2a02a"}
Nov 02 17:44:44.072570 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:44.072494    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f" podStartSLOduration=2.072470275 podStartE2EDuration="2.072470275s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:44.063117239 +0000 UTC m=+900.554180401" watchObservedRunningTime="2026-11-02 17:44:44.072470275 +0000 UTC m=+900.563533438"
Nov 02 17:44:44.077233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:44.077181    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:44.848278 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:44.848238    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"]
Nov 02 17:44:44.857177 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:44.857146    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"
Nov 02 17:44:44.900662 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:44.900628    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"]
Nov 02 17:44:44.951601 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:44.951565    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd769fa3-1722-4b6d-85a2-88602431a9aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6eed92d-rnwjw\" (UID: \"cd769fa3-1722-4b6d-85a2-88602431a9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"
Nov 02 17:44:44.951761 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:44.951627    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bkpm2\" (UniqueName: \"kubernetes.io/projected/cd769fa3-1722-4b6d-85a2-88602431a9aa-kube-api-access-bkpm2\") pod \"maas-api-revoke-sub-f6eed92d-rnwjw\" (UID: \"cd769fa3-1722-4b6d-85a2-88602431a9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"
Nov 02 17:44:44.951971 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:44.951947    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd769fa3-1722-4b6d-85a2-88602431a9aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6eed92d-rnwjw\" (UID: \"cd769fa3-1722-4b6d-85a2-88602431a9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"
Nov 02 17:44:45.039704 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:45.039221    4456 generic.go:356] "Generic (PLEG): container finished" podID="50522edc-3a5e-4c72-a147-8b99ba6c9dc1" containerID="883eff289f03da617853c7e0d1141e259b3c0a03e1b0569a985e9ed7312a83d2" exitCode=0
Nov 02 17:44:45.039704 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:45.039502    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f" event={"ID":"50522edc-3a5e-4c72-a147-8b99ba6c9dc1","Type":"ContainerDied","Data":"883eff289f03da617853c7e0d1141e259b3c0a03e1b0569a985e9ed7312a83d2"}
Nov 02 17:44:45.062842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:45.055009    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd769fa3-1722-4b6d-85a2-88602431a9aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6eed92d-rnwjw\" (UID: \"cd769fa3-1722-4b6d-85a2-88602431a9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"
Nov 02 17:44:45.062842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:45.055165    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd769fa3-1722-4b6d-85a2-88602431a9aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6eed92d-rnwjw\" (UID: \"cd769fa3-1722-4b6d-85a2-88602431a9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"
Nov 02 17:44:45.062842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:45.055223    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bkpm2\" (UniqueName: \"kubernetes.io/projected/cd769fa3-1722-4b6d-85a2-88602431a9aa-kube-api-access-bkpm2\") pod \"maas-api-revoke-sub-f6eed92d-rnwjw\" (UID: \"cd769fa3-1722-4b6d-85a2-88602431a9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"
Nov 02 17:44:45.062842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:45.056320    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd769fa3-1722-4b6d-85a2-88602431a9aa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f6eed92d-rnwjw\" (UID: \"cd769fa3-1722-4b6d-85a2-88602431a9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"
Nov 02 17:44:45.074838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:45.064707    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd769fa3-1722-4b6d-85a2-88602431a9aa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f6eed92d-rnwjw\" (UID: \"cd769fa3-1722-4b6d-85a2-88602431a9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"
Nov 02 17:44:45.124002 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:45.117948    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bkpm2\" (UniqueName: \"kubernetes.io/projected/cd769fa3-1722-4b6d-85a2-88602431a9aa-kube-api-access-bkpm2\") pod \"maas-api-revoke-sub-f6eed92d-rnwjw\" (UID: \"cd769fa3-1722-4b6d-85a2-88602431a9aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"
Nov 02 17:44:45.202847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:45.199357    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"
Nov 02 17:44:46.282854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:46.260102    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:46.704839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:46.691923    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:46.704839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:46.692220    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:44:46.879992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:46.879945    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"]
Nov 02 17:44:46.987265 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:46.987142    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:47.001996 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.001939    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:47.010125 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.010070    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:47.027890 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.021035    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 17:44:47.100247 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.100211    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"]
Nov 02 17:44:47.112948 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:44:47.102044    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:44:47.113215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.102756    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:47.156042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.156005    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 17:44:47.217903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.217869    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"]
Nov 02 17:44:47.268392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.265223    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:47.295101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.268549    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:47.295101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.285407    4456 patch_prober.go:29] interesting pod/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:44:47.295101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.285482    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr" podUID="fb933398-5485-4600-a38e-640531c54137" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:47.295101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.292251    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:44:47.340184 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.309722    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:47.588156 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.588057    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"]
Nov 02 17:44:47.767955 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.690398    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:47.806437 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.806398    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:47.814844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.806778    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:47.818963 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.818925    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:47.824022 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.823980    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fgjrd\" (UniqueName: \"kubernetes.io/projected/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-kube-api-access-fgjrd\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.001543 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.001507    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.001897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.001869    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.012106 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.012072    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.016846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.012313    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.237918 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.695966    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:44:48.238646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.011970    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:48.277897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.001392    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 17:44:48.277897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:47.695905    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:48.277897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.017115    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:48.277897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.031011    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:48.277897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.268495    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:48.277897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.240168    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:44:48.277897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.268695    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:44:48.277897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.268761    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:44:48.277897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.268801    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:44:48.277897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.268861    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:44:48.277897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.268897    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:44:48.292869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.174111    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:48.292869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.286371    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:44:48.317736 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.317695    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 17:44:48.382105 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.361666    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 17:44:48.382728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.382698    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:44:48.411842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.386477    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-px957\" (UniqueName: \"kubernetes.io/projected/c5f3dfbb-c7df-4c2b-8531-983517c22929-kube-api-access-px957\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:48.412151 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.412116    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:48.412309 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.412290    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.412452 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.412432    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.412608 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.412589    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:48.412856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.412835    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f3dfbb-c7df-4c2b-8531-983517c22929-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:48.413165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.413142    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.413326 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.413310    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.413615 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.413596    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.413742 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.413727    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.414014 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.413990    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:48.495843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.479024    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:48.495843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.479105    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:48.495843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.479136    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:48.495843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.479178    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fgjrd\" (UniqueName: \"kubernetes.io/projected/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-kube-api-access-fgjrd\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.495843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.479216    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.495843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.479254    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:48.495843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.479302    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.495843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.434844    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-data\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.495843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.435082    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-workload-certs\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.495843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.435312    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-workload-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.495843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.492765    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-credential-socket\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.587169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.583964    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-podinfo\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.681425 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.661788    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istiod-ca-cert\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:48.767257 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:48.761004    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"]
Nov 02 17:44:49.007696 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.007650    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 02 17:44:49.059749 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.053672    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-envoy\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:49.199843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.126337    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-token\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:49.199843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.181216    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:49.199843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.181302    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:49.199843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.181350    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:49.199843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.181385    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:49.199843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.181469    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:49.199843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.181545    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-px957\" (UniqueName: \"kubernetes.io/projected/c5f3dfbb-c7df-4c2b-8531-983517c22929-kube-api-access-px957\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:49.199843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.181589    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:49.199843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.181676    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:49.199843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.181774    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f3dfbb-c7df-4c2b-8531-983517c22929-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:49.265933 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.265798    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:49.279882 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.270118    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:49.405842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.280890    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-credential-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:49.405842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.393922    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:44:49.405842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.394248    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 17:44:49.405842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.394488    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:44:49.405842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.394688    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 17:44:49.405842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.396905    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-workload-certs\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:49.441845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.434794    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-data\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:49.557886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.546258    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 17:44:49.557886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.546632    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 17:44:49.557886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.555336    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 17:44:49.572844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.568995    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-workload-socket\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:49.730612 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.691782    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-podinfo\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:49.730922 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.701130    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-envoy\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:49.732650 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:49.732621    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fgjrd\" (UniqueName: \"kubernetes.io/projected/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-kube-api-access-fgjrd\") pod \"e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") " pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:50.127840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.120142    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f3dfbb-c7df-4c2b-8531-983517c22929-istiod-ca-cert\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:50.274841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.211519    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-px957\" (UniqueName: \"kubernetes.io/projected/c5f3dfbb-c7df-4c2b-8531-983517c22929-kube-api-access-px957\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:50.318938 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.296494    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:50.318938 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.305447    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"
Nov 02 17:44:50.395839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.390949    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:44:50.426668 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.419128    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:50.439840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.419185    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:44:50.447848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.419204    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 17:44:50.455835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.419233    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:44:50.467063 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.419272    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 17:44:50.474842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.419316    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:50.578422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.425164    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-token\") pod \"e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") " pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:50.586952 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.426264    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:44:50.672978 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.549108    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:50.673398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.549174    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 17:44:50.673733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.577057    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:50.674269 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.674223    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:50.770843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.745915    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:50.774840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.773415    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:50.888895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.888685    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-kube-api-access-6v9h9\" (UniqueName: \"kubernetes.io/projected/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-kube-api-access-6v9h9\") pod \"50522edc-3a5e-4c72-a147-8b99ba6c9dc1\" (UID: \"50522edc-3a5e-4c72-a147-8b99ba6c9dc1\") "
Nov 02 17:44:50.888895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.888802    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-maas-api-openshift-ca\") pod \"50522edc-3a5e-4c72-a147-8b99ba6c9dc1\" (UID: \"50522edc-3a5e-4c72-a147-8b99ba6c9dc1\") "
Nov 02 17:44:50.889126 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.888979    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-maas-api-xks-ca\") pod \"50522edc-3a5e-4c72-a147-8b99ba6c9dc1\" (UID: \"50522edc-3a5e-4c72-a147-8b99ba6c9dc1\") "
Nov 02 17:44:50.940029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.939972    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:50.941056 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.941015    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:44:50.941289 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:50.941262    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:51.320841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:51.317351    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-maas-api-openshift-ca" pod "50522edc-3a5e-4c72-a147-8b99ba6c9dc1" (UID: "50522edc-3a5e-4c72-a147-8b99ba6c9dc1"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:51.345766 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:51.345731    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:51.450054 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:51.419926    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:51.450054 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:51.430134    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:44:51.465330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:51.463955    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:51.465659 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:51.465609    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:51.630893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:51.628506    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:44:51.630893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:51.629014    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:51.631144 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:51.630879    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:51.658128 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:44:51.646742    4456 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.634s"
Nov 02 17:44:51.658128 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:51.646780    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw" event={"ID":"cd769fa3-1722-4b6d-85a2-88602431a9aa","Type":"ContainerStarted","Data":"8308dd7023be0c69be5934e7726b109826fcbf8629da5d876618cff5d5ee8027"}
Nov 02 17:44:51.845771 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:51.845727    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 17:44:51.847322 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:51.847291    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:52.076116 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.076067    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:44:52.083131 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.083101    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:52.166146 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.166107    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-maas-api-xks-ca" pod "50522edc-3a5e-4c72-a147-8b99ba6c9dc1" (UID: "50522edc-3a5e-4c72-a147-8b99ba6c9dc1"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:44:52.252585 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.243241    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:52.280528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.278690    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"
Nov 02 17:44:52.291619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.291583    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:52.292654 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.292609    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:52.292850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.292798    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:52.327877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.327349    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-kube-api-access-6v9h9" pod "50522edc-3a5e-4c72-a147-8b99ba6c9dc1" (UID: "50522edc-3a5e-4c72-a147-8b99ba6c9dc1"). InnerVolumeSpecName "kube-api-access-6v9h9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:52.354832 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.354785    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6v9h9\" (UniqueName: \"kubernetes.io/projected/50522edc-3a5e-4c72-a147-8b99ba6c9dc1-kube-api-access-6v9h9\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:52.360194 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.360159    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f" event={"ID":"50522edc-3a5e-4c72-a147-8b99ba6c9dc1","Type":"ContainerDied","Data":"8cbcf7dd63ad35e2b5c304de7d01c4a0fd98f322debfd9d49468599f30c2a02a"}
Nov 02 17:44:52.360398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.360382    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8cbcf7dd63ad35e2b5c304de7d01c4a0fd98f322debfd9d49468599f30c2a02a"
Nov 02 17:44:52.360526 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.360515    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:44:52.360765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.360749    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:44:52.360969 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.360955    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:44:52.361087 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.361077    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:44:52.369936 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.361516    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:44:52.369936 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.361557    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:44:52.369936 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.361573    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw" event={"ID":"cd769fa3-1722-4b6d-85a2-88602431a9aa","Type":"ContainerStarted","Data":"22d60d1711c3fcaa2777f36c5f4f4bc423ed26bbef3e3b1da5fd1d450a05e744"}
Nov 02 17:44:52.369936 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.361625    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:44:52.369936 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.361653    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:44:52.369936 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.361681    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:44:52.407850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.407799    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"]
Nov 02 17:44:52.412571 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.412512    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw" podStartSLOduration=8.412494328 podStartE2EDuration="8.412494328s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:52.352636803 +0000 UTC m=+908.843699966" watchObservedRunningTime="2026-11-02 17:44:52.412494328 +0000 UTC m=+908.903557489"
Nov 02 17:44:52.475441 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.475408    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"]
Nov 02 17:44:52.498315 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:44:52.498272    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf072ed0e_d8c7_48ea_973b_9b09b1b1f0d3.slice/crio-cdd80a3f67466a8e118c64c734d288d17970ba233eefdf4a96fcb78951cac647 WatchSource:0}: Error finding container cdd80a3f67466a8e118c64c734d288d17970ba233eefdf4a96fcb78951cac647: Status 404 returned error can't find the container with id cdd80a3f67466a8e118c64c734d288d17970ba233eefdf4a96fcb78951cac647
Nov 02 17:44:52.505679 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.505615    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:52.505993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.505970    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:52.506556 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:52.506532    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:44:53.000368 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.000290    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"]
Nov 02 17:44:53.001082 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.001059    4456 cpu_manager.go:414] "containerMap: removing container" podUID="50522edc-3a5e-4c72-a147-8b99ba6c9dc1" containerName="revoke-keys"
Nov 02 17:44:53.001082 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.001079    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="50522edc-3a5e-4c72-a147-8b99ba6c9dc1" containerName="revoke-keys"
Nov 02 17:44:53.001258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.001218    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="50522edc-3a5e-4c72-a147-8b99ba6c9dc1" containerName="revoke-keys"
Nov 02 17:44:53.005179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.005157    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"
Nov 02 17:44:53.025626 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.025592    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"]
Nov 02 17:44:53.070474 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.070425    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa7b357-2411-42dd-837a-cc36d6133378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b398b4-9w4dl\" (UID: \"efa7b357-2411-42dd-837a-cc36d6133378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"
Nov 02 17:44:53.070651 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.070550    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2dcfb\" (UniqueName: \"kubernetes.io/projected/efa7b357-2411-42dd-837a-cc36d6133378-kube-api-access-2dcfb\") pod \"maas-api-revoke-sub-28b398b4-9w4dl\" (UID: \"efa7b357-2411-42dd-837a-cc36d6133378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"
Nov 02 17:44:53.070706 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.070671    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa7b357-2411-42dd-837a-cc36d6133378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b398b4-9w4dl\" (UID: \"efa7b357-2411-42dd-837a-cc36d6133378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"
Nov 02 17:44:53.177925 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.174917    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa7b357-2411-42dd-837a-cc36d6133378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b398b4-9w4dl\" (UID: \"efa7b357-2411-42dd-837a-cc36d6133378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"
Nov 02 17:44:53.177925 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.175167    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa7b357-2411-42dd-837a-cc36d6133378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b398b4-9w4dl\" (UID: \"efa7b357-2411-42dd-837a-cc36d6133378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"
Nov 02 17:44:53.177925 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.175242    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2dcfb\" (UniqueName: \"kubernetes.io/projected/efa7b357-2411-42dd-837a-cc36d6133378-kube-api-access-2dcfb\") pod \"maas-api-revoke-sub-28b398b4-9w4dl\" (UID: \"efa7b357-2411-42dd-837a-cc36d6133378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"
Nov 02 17:44:53.178699 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.178666    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa7b357-2411-42dd-837a-cc36d6133378-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-28b398b4-9w4dl\" (UID: \"efa7b357-2411-42dd-837a-cc36d6133378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"
Nov 02 17:44:53.223839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.214162    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa7b357-2411-42dd-837a-cc36d6133378-maas-api-xks-ca\") pod \"maas-api-revoke-sub-28b398b4-9w4dl\" (UID: \"efa7b357-2411-42dd-837a-cc36d6133378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"
Nov 02 17:44:53.235841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.232805    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2dcfb\" (UniqueName: \"kubernetes.io/projected/efa7b357-2411-42dd-837a-cc36d6133378-kube-api-access-2dcfb\") pod \"maas-api-revoke-sub-28b398b4-9w4dl\" (UID: \"efa7b357-2411-42dd-837a-cc36d6133378\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"
Nov 02 17:44:53.327897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.326889    4456 generic.go:356] "Generic (PLEG): container finished" podID="cd769fa3-1722-4b6d-85a2-88602431a9aa" containerID="22d60d1711c3fcaa2777f36c5f4f4bc423ed26bbef3e3b1da5fd1d450a05e744" exitCode=0
Nov 02 17:44:53.327897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.327037    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw" event={"ID":"cd769fa3-1722-4b6d-85a2-88602431a9aa","Type":"ContainerDied","Data":"22d60d1711c3fcaa2777f36c5f4f4bc423ed26bbef3e3b1da5fd1d450a05e744"}
Nov 02 17:44:53.333502 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.329704    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44" event={"ID":"c5f3dfbb-c7df-4c2b-8531-983517c22929","Type":"ContainerStarted","Data":"88422983dda55c734dfd03b064a332cab84b26aa5c593416425202c7dae4255f"}
Nov 02 17:44:53.333502 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.329763    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44" event={"ID":"c5f3dfbb-c7df-4c2b-8531-983517c22929","Type":"ContainerStarted","Data":"abbab27e711230b0e1ae5e7c024f18bc32de7047ce2c61096ff9211a8ee86334"}
Nov 02 17:44:53.333502 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.332294    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"
Nov 02 17:44:53.333502 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.333467    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v" event={"ID":"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3","Type":"ContainerStarted","Data":"59f84b293d5554412b761fd7de2e57d2b60039eae14a8b53dbeaa239477d67bd"}
Nov 02 17:44:53.333854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.333520    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v" event={"ID":"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3","Type":"ContainerStarted","Data":"cdd80a3f67466a8e118c64c734d288d17970ba233eefdf4a96fcb78951cac647"}
Nov 02 17:44:53.559841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.559419    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v" podStartSLOduration=7.559395667 podStartE2EDuration="7.559395667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:53.479286722 +0000 UTC m=+909.970349920" watchObservedRunningTime="2026-11-02 17:44:53.559395667 +0000 UTC m=+910.050458828"
Nov 02 17:44:53.610045 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.571711    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44" podStartSLOduration=7.571691747 podStartE2EDuration="7.571691747s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:44:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:44:53.554380291 +0000 UTC m=+910.045443565" watchObservedRunningTime="2026-11-02 17:44:53.571691747 +0000 UTC m=+910.062754909"
Nov 02 17:44:53.825986 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.825937    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:44:53.842879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.841974    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 17:44:53.842879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:53.842063    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44" podUID="c5f3dfbb-c7df-4c2b-8531-983517c22929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 17:44:54.054900 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:54.053191    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"]
Nov 02 17:44:54.056057 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:44:54.056028    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podefa7b357_2411_42dd_837a_cc36d6133378.slice/crio-1bc2dfe1ad50e9177392a394e06936ea508070f0a1ccd0d6337958cc94f67cdf WatchSource:0}: Error finding container 1bc2dfe1ad50e9177392a394e06936ea508070f0a1ccd0d6337958cc94f67cdf: Status 404 returned error can't find the container with id 1bc2dfe1ad50e9177392a394e06936ea508070f0a1ccd0d6337958cc94f67cdf
Nov 02 17:44:54.217456 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:54.217415    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:54.299762 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:54.299674    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:44:54.300122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:54.300098    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 17:44:54.305954 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:54.305915    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v" podUID="f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 17:44:54.463203 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:54.463167    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl" event={"ID":"efa7b357-2411-42dd-837a-cc36d6133378","Type":"ContainerStarted","Data":"1bc2dfe1ad50e9177392a394e06936ea508070f0a1ccd0d6337958cc94f67cdf"}
Nov 02 17:44:54.822842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:54.811284    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 17:44:54.822842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:54.811348    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44" podUID="c5f3dfbb-c7df-4c2b-8531-983517c22929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 17:44:54.860921 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:54.860852    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"
Nov 02 17:44:54.969802 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:54.969767    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cd769fa3-1722-4b6d-85a2-88602431a9aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd769fa3-1722-4b6d-85a2-88602431a9aa-maas-api-openshift-ca\") pod \"cd769fa3-1722-4b6d-85a2-88602431a9aa\" (UID: \"cd769fa3-1722-4b6d-85a2-88602431a9aa\") "
Nov 02 17:44:54.969969 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:54.969837    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cd769fa3-1722-4b6d-85a2-88602431a9aa-kube-api-access-bkpm2\" (UniqueName: \"kubernetes.io/projected/cd769fa3-1722-4b6d-85a2-88602431a9aa-kube-api-access-bkpm2\") pod \"cd769fa3-1722-4b6d-85a2-88602431a9aa\" (UID: \"cd769fa3-1722-4b6d-85a2-88602431a9aa\") "
Nov 02 17:44:54.970030 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:54.970006    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cd769fa3-1722-4b6d-85a2-88602431a9aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd769fa3-1722-4b6d-85a2-88602431a9aa-maas-api-xks-ca\") pod \"cd769fa3-1722-4b6d-85a2-88602431a9aa\" (UID: \"cd769fa3-1722-4b6d-85a2-88602431a9aa\") "
Nov 02 17:44:54.971362 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:54.971303    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cd769fa3-1722-4b6d-85a2-88602431a9aa-maas-api-openshift-ca" pod "cd769fa3-1722-4b6d-85a2-88602431a9aa" (UID: "cd769fa3-1722-4b6d-85a2-88602431a9aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:54.972875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:54.972854    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cd769fa3-1722-4b6d-85a2-88602431a9aa-maas-api-xks-ca" pod "cd769fa3-1722-4b6d-85a2-88602431a9aa" (UID: "cd769fa3-1722-4b6d-85a2-88602431a9aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:44:54.974295 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:54.974263    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cd769fa3-1722-4b6d-85a2-88602431a9aa-kube-api-access-bkpm2" pod "cd769fa3-1722-4b6d-85a2-88602431a9aa" (UID: "cd769fa3-1722-4b6d-85a2-88602431a9aa"). InnerVolumeSpecName "kube-api-access-bkpm2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:55.072009 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.071975    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cd769fa3-1722-4b6d-85a2-88602431a9aa-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:55.072009 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.072003    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bkpm2\" (UniqueName: \"kubernetes.io/projected/cd769fa3-1722-4b6d-85a2-88602431a9aa-kube-api-access-bkpm2\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:55.072009 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.072014    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cd769fa3-1722-4b6d-85a2-88602431a9aa-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:55.257892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.256752    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"]
Nov 02 17:44:55.258126 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.257941    4456 cpu_manager.go:414] "containerMap: removing container" podUID="cd769fa3-1722-4b6d-85a2-88602431a9aa" containerName="revoke-keys"
Nov 02 17:44:55.258126 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.257962    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cd769fa3-1722-4b6d-85a2-88602431a9aa" containerName="revoke-keys"
Nov 02 17:44:55.258216 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.258137    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="cd769fa3-1722-4b6d-85a2-88602431a9aa" containerName="revoke-keys"
Nov 02 17:44:55.264872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.262804    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"
Nov 02 17:44:55.264872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.264047    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"]
Nov 02 17:44:55.340874 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.322734    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 17:44:55.340874 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.322795    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v" podUID="f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 17:44:55.378559 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.378520    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5hqdz\" (UniqueName: \"kubernetes.io/projected/84a0333b-4b46-4853-af3c-a5aeb904a877-kube-api-access-5hqdz\") pod \"maas-api-revoke-sub-e9ffd4bb-8jqvf\" (UID: \"84a0333b-4b46-4853-af3c-a5aeb904a877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"
Nov 02 17:44:55.378713 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.378588    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a0333b-4b46-4853-af3c-a5aeb904a877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9ffd4bb-8jqvf\" (UID: \"84a0333b-4b46-4853-af3c-a5aeb904a877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"
Nov 02 17:44:55.378798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.378740    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a0333b-4b46-4853-af3c-a5aeb904a877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9ffd4bb-8jqvf\" (UID: \"84a0333b-4b46-4853-af3c-a5aeb904a877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"
Nov 02 17:44:55.489841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.480170    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a0333b-4b46-4853-af3c-a5aeb904a877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9ffd4bb-8jqvf\" (UID: \"84a0333b-4b46-4853-af3c-a5aeb904a877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"
Nov 02 17:44:55.489841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.480476    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5hqdz\" (UniqueName: \"kubernetes.io/projected/84a0333b-4b46-4853-af3c-a5aeb904a877-kube-api-access-5hqdz\") pod \"maas-api-revoke-sub-e9ffd4bb-8jqvf\" (UID: \"84a0333b-4b46-4853-af3c-a5aeb904a877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"
Nov 02 17:44:55.489841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.480505    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a0333b-4b46-4853-af3c-a5aeb904a877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9ffd4bb-8jqvf\" (UID: \"84a0333b-4b46-4853-af3c-a5aeb904a877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"
Nov 02 17:44:55.513936 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.491296    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a0333b-4b46-4853-af3c-a5aeb904a877-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e9ffd4bb-8jqvf\" (UID: \"84a0333b-4b46-4853-af3c-a5aeb904a877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"
Nov 02 17:44:55.542883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.526966    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5hqdz\" (UniqueName: \"kubernetes.io/projected/84a0333b-4b46-4853-af3c-a5aeb904a877-kube-api-access-5hqdz\") pod \"maas-api-revoke-sub-e9ffd4bb-8jqvf\" (UID: \"84a0333b-4b46-4853-af3c-a5aeb904a877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"
Nov 02 17:44:55.546888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.546860    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"
Nov 02 17:44:55.548455 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.547106    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw" event={"ID":"cd769fa3-1722-4b6d-85a2-88602431a9aa","Type":"ContainerDied","Data":"8308dd7023be0c69be5934e7726b109826fcbf8629da5d876618cff5d5ee8027"}
Nov 02 17:44:55.548557 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.548487    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8308dd7023be0c69be5934e7726b109826fcbf8629da5d876618cff5d5ee8027"
Nov 02 17:44:55.552361 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.552323    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a0333b-4b46-4853-af3c-a5aeb904a877-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e9ffd4bb-8jqvf\" (UID: \"84a0333b-4b46-4853-af3c-a5aeb904a877\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"
Nov 02 17:44:55.554030 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.554005    4456 generic.go:356] "Generic (PLEG): container finished" podID="efa7b357-2411-42dd-837a-cc36d6133378" containerID="6b3136dd6effce815920358b5ef4e25d78f4e6f73f6587a51eadbdc4f33c175d" exitCode=0
Nov 02 17:44:55.554188 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.554166    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl" event={"ID":"efa7b357-2411-42dd-837a-cc36d6133378","Type":"ContainerDied","Data":"6b3136dd6effce815920358b5ef4e25d78f4e6f73f6587a51eadbdc4f33c175d"}
Nov 02 17:44:55.579571 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.579552    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"
Nov 02 17:44:55.865984 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.864442    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 17:44:55.865984 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:55.864503    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44" podUID="c5f3dfbb-c7df-4c2b-8531-983517c22929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 17:44:56.048890 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:56.044285    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"]
Nov 02 17:44:56.266092 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:56.266057    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:56.302849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:56.299234    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 17:44:56.302849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:56.299293    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v" podUID="f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 17:44:56.591892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:56.587417    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf" event={"ID":"84a0333b-4b46-4853-af3c-a5aeb904a877","Type":"ContainerStarted","Data":"b210388e7881dcc7a81c830a64320bf0a634850e1337e9beac4af687b86d8e8d"}
Nov 02 17:44:56.804261 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:56.804221    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 17:44:56.804533 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:56.804494    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44" podUID="c5f3dfbb-c7df-4c2b-8531-983517c22929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 17:44:57.159339 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.159308    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"
Nov 02 17:44:57.267840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.267010    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/efa7b357-2411-42dd-837a-cc36d6133378-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa7b357-2411-42dd-837a-cc36d6133378-maas-api-xks-ca\") pod \"efa7b357-2411-42dd-837a-cc36d6133378\" (UID: \"efa7b357-2411-42dd-837a-cc36d6133378\") "
Nov 02 17:44:57.267840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.267120    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/efa7b357-2411-42dd-837a-cc36d6133378-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa7b357-2411-42dd-837a-cc36d6133378-maas-api-openshift-ca\") pod \"efa7b357-2411-42dd-837a-cc36d6133378\" (UID: \"efa7b357-2411-42dd-837a-cc36d6133378\") "
Nov 02 17:44:57.267840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.267218    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/efa7b357-2411-42dd-837a-cc36d6133378-kube-api-access-2dcfb\" (UniqueName: \"kubernetes.io/projected/efa7b357-2411-42dd-837a-cc36d6133378-kube-api-access-2dcfb\") pod \"efa7b357-2411-42dd-837a-cc36d6133378\" (UID: \"efa7b357-2411-42dd-837a-cc36d6133378\") "
Nov 02 17:44:57.269103 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.269063    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/efa7b357-2411-42dd-837a-cc36d6133378-maas-api-openshift-ca" pod "efa7b357-2411-42dd-837a-cc36d6133378" (UID: "efa7b357-2411-42dd-837a-cc36d6133378"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:57.276971 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.276939    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/efa7b357-2411-42dd-837a-cc36d6133378-maas-api-xks-ca" pod "efa7b357-2411-42dd-837a-cc36d6133378" (UID: "efa7b357-2411-42dd-837a-cc36d6133378"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:44:57.277934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.277909    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/efa7b357-2411-42dd-837a-cc36d6133378-kube-api-access-2dcfb" pod "efa7b357-2411-42dd-837a-cc36d6133378" (UID: "efa7b357-2411-42dd-837a-cc36d6133378"). InnerVolumeSpecName "kube-api-access-2dcfb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:57.322975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.300226    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 17:44:57.322975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.300283    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v" podUID="f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 17:44:57.369644 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.369612    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/efa7b357-2411-42dd-837a-cc36d6133378-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:57.369644 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.369646    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/efa7b357-2411-42dd-837a-cc36d6133378-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:57.369889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.369661    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2dcfb\" (UniqueName: \"kubernetes.io/projected/efa7b357-2411-42dd-837a-cc36d6133378-kube-api-access-2dcfb\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:57.657986 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.657898    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"
Nov 02 17:44:57.661714 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.661681    4456 generic.go:356] "Generic (PLEG): container finished" podID="84a0333b-4b46-4853-af3c-a5aeb904a877" containerID="e24cc9891fc9fd824c91fbfbff91184fee256eed39286c80691938f48cf2cd25" exitCode=0
Nov 02 17:44:57.663085 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.663058    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl" event={"ID":"efa7b357-2411-42dd-837a-cc36d6133378","Type":"ContainerDied","Data":"1bc2dfe1ad50e9177392a394e06936ea508070f0a1ccd0d6337958cc94f67cdf"}
Nov 02 17:44:57.663164 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.663115    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1bc2dfe1ad50e9177392a394e06936ea508070f0a1ccd0d6337958cc94f67cdf"
Nov 02 17:44:57.663164 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.663129    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf" event={"ID":"84a0333b-4b46-4853-af3c-a5aeb904a877","Type":"ContainerDied","Data":"e24cc9891fc9fd824c91fbfbff91184fee256eed39286c80691938f48cf2cd25"}
Nov 02 17:44:57.804849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.804294    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 17:44:57.804849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:57.804358    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44" podUID="c5f3dfbb-c7df-4c2b-8531-983517c22929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 17:44:58.200699 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:58.199329    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:44:58.315842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:58.314951    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 17:44:58.315842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:58.315017    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v" podUID="f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 17:44:58.802897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:58.800450    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused" start-of-body=
Nov 02 17:44:58.802897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:58.800511    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44" podUID="c5f3dfbb-c7df-4c2b-8531-983517c22929" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.100:15021/healthz/ready\": dial tcp 10.134.0.100:15021: connect: connection refused"
Nov 02 17:44:58.963509 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:58.963484    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"
Nov 02 17:44:59.036988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:59.035564    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/84a0333b-4b46-4853-af3c-a5aeb904a877-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a0333b-4b46-4853-af3c-a5aeb904a877-maas-api-xks-ca\") pod \"84a0333b-4b46-4853-af3c-a5aeb904a877\" (UID: \"84a0333b-4b46-4853-af3c-a5aeb904a877\") "
Nov 02 17:44:59.036988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:59.035915    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/84a0333b-4b46-4853-af3c-a5aeb904a877-kube-api-access-5hqdz\" (UniqueName: \"kubernetes.io/projected/84a0333b-4b46-4853-af3c-a5aeb904a877-kube-api-access-5hqdz\") pod \"84a0333b-4b46-4853-af3c-a5aeb904a877\" (UID: \"84a0333b-4b46-4853-af3c-a5aeb904a877\") "
Nov 02 17:44:59.036988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:59.036051    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/84a0333b-4b46-4853-af3c-a5aeb904a877-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a0333b-4b46-4853-af3c-a5aeb904a877-maas-api-openshift-ca\") pod \"84a0333b-4b46-4853-af3c-a5aeb904a877\" (UID: \"84a0333b-4b46-4853-af3c-a5aeb904a877\") "
Nov 02 17:44:59.037834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:59.037511    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/84a0333b-4b46-4853-af3c-a5aeb904a877-maas-api-openshift-ca" pod "84a0333b-4b46-4853-af3c-a5aeb904a877" (UID: "84a0333b-4b46-4853-af3c-a5aeb904a877"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:44:59.051232 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:59.051202    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/84a0333b-4b46-4853-af3c-a5aeb904a877-kube-api-access-5hqdz" pod "84a0333b-4b46-4853-af3c-a5aeb904a877" (UID: "84a0333b-4b46-4853-af3c-a5aeb904a877"). InnerVolumeSpecName "kube-api-access-5hqdz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:44:59.053084 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:59.053043    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/84a0333b-4b46-4853-af3c-a5aeb904a877-maas-api-xks-ca" pod "84a0333b-4b46-4853-af3c-a5aeb904a877" (UID: "84a0333b-4b46-4853-af3c-a5aeb904a877"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:44:59.138349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:59.138313    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5hqdz\" (UniqueName: \"kubernetes.io/projected/84a0333b-4b46-4853-af3c-a5aeb904a877-kube-api-access-5hqdz\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:59.138349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:59.138343    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/84a0333b-4b46-4853-af3c-a5aeb904a877-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:59.138349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:59.138353    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/84a0333b-4b46-4853-af3c-a5aeb904a877-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:44:59.297608 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:59.297567    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused" start-of-body=
Nov 02 17:44:59.298125 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:59.297619    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v" podUID="f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.99:15021/healthz/ready\": dial tcp 10.134.0.99:15021: connect: connection refused"
Nov 02 17:44:59.691626 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:59.691590    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"
Nov 02 17:44:59.691626 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:59.691612    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf" event={"ID":"84a0333b-4b46-4853-af3c-a5aeb904a877","Type":"ContainerDied","Data":"b210388e7881dcc7a81c830a64320bf0a634850e1337e9beac4af687b86d8e8d"}
Nov 02 17:44:59.691865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:44:59.691661    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b210388e7881dcc7a81c830a64320bf0a634850e1337e9beac4af687b86d8e8d"
Nov 02 17:45:00.081249 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.081210    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:00.192783 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.192750    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"]
Nov 02 17:45:00.193792 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.193770    4456 cpu_manager.go:414] "containerMap: removing container" podUID="efa7b357-2411-42dd-837a-cc36d6133378" containerName="revoke-keys"
Nov 02 17:45:00.193896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.193795    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="efa7b357-2411-42dd-837a-cc36d6133378" containerName="revoke-keys"
Nov 02 17:45:00.193896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.193843    4456 cpu_manager.go:414] "containerMap: removing container" podUID="84a0333b-4b46-4853-af3c-a5aeb904a877" containerName="revoke-keys"
Nov 02 17:45:00.193896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.193852    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="84a0333b-4b46-4853-af3c-a5aeb904a877" containerName="revoke-keys"
Nov 02 17:45:00.194044 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.194030    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="efa7b357-2411-42dd-837a-cc36d6133378" containerName="revoke-keys"
Nov 02 17:45:00.194091 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.194052    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="84a0333b-4b46-4853-af3c-a5aeb904a877" containerName="revoke-keys"
Nov 02 17:45:00.198528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.198506    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"
Nov 02 17:45:00.199891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.199869    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"]
Nov 02 17:45:00.205051 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.205029    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"
Nov 02 17:45:00.211840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.209208    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"]
Nov 02 17:45:00.219862 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.219054    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"
Nov 02 17:45:00.256713 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.242076    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:45:00.270464 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.269870    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"]
Nov 02 17:45:00.272233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.272207    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lr56b\" (UniqueName: \"kubernetes.io/projected/24287229-12c3-402a-a20a-ad35eaa2ce51-kube-api-access-lr56b\") pod \"maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l\" (UID: \"24287229-12c3-402a-a20a-ad35eaa2ce51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"
Nov 02 17:45:00.272472 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.272449    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hzpd7\" (UniqueName: \"kubernetes.io/projected/f3799555-a5eb-44d8-99c8-6df387241cf5-kube-api-access-hzpd7\") pod \"maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp\" (UID: \"f3799555-a5eb-44d8-99c8-6df387241cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"
Nov 02 17:45:00.272593 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.272577    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dfpxb\" (UniqueName: \"kubernetes.io/projected/f60eecce-a69b-4055-8c35-270689f1cdbd-kube-api-access-dfpxb\") pod \"maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r\" (UID: \"f60eecce-a69b-4055-8c35-270689f1cdbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"
Nov 02 17:45:00.272847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.272803    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60eecce-a69b-4055-8c35-270689f1cdbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r\" (UID: \"f60eecce-a69b-4055-8c35-270689f1cdbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"
Nov 02 17:45:00.272998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.272981    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3799555-a5eb-44d8-99c8-6df387241cf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp\" (UID: \"f3799555-a5eb-44d8-99c8-6df387241cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"
Nov 02 17:45:00.273212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.273194    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24287229-12c3-402a-a20a-ad35eaa2ce51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l\" (UID: \"24287229-12c3-402a-a20a-ad35eaa2ce51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"
Nov 02 17:45:00.274535 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.274477    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"]
Nov 02 17:45:00.285256 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.285232    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"]
Nov 02 17:45:00.292350 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.292329    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"
Nov 02 17:45:00.299032 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.298961    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"]
Nov 02 17:45:00.308619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.308556    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"]
Nov 02 17:45:00.366862 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.362150    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:45:00.366862 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.363313    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:45:00.366862 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.364420    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:45:00.374339 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.374313    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60eecce-a69b-4055-8c35-270689f1cdbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r\" (UID: \"f60eecce-a69b-4055-8c35-270689f1cdbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"
Nov 02 17:45:00.374474 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.374363    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dxwbm\" (UniqueName: \"kubernetes.io/projected/e703782b-0ae2-4e44-8a37-cbd921b3ebec-kube-api-access-dxwbm\") pod \"maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn\" (UID: \"e703782b-0ae2-4e44-8a37-cbd921b3ebec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"
Nov 02 17:45:00.374474 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.374401    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3799555-a5eb-44d8-99c8-6df387241cf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp\" (UID: \"f3799555-a5eb-44d8-99c8-6df387241cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"
Nov 02 17:45:00.374609 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.374515    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24287229-12c3-402a-a20a-ad35eaa2ce51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l\" (UID: \"24287229-12c3-402a-a20a-ad35eaa2ce51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"
Nov 02 17:45:00.374679 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.374609    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lr56b\" (UniqueName: \"kubernetes.io/projected/24287229-12c3-402a-a20a-ad35eaa2ce51-kube-api-access-lr56b\") pod \"maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l\" (UID: \"24287229-12c3-402a-a20a-ad35eaa2ce51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"
Nov 02 17:45:00.374679 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.374639    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e703782b-0ae2-4e44-8a37-cbd921b3ebec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn\" (UID: \"e703782b-0ae2-4e44-8a37-cbd921b3ebec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"
Nov 02 17:45:00.374790 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.374681    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hzpd7\" (UniqueName: \"kubernetes.io/projected/f3799555-a5eb-44d8-99c8-6df387241cf5-kube-api-access-hzpd7\") pod \"maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp\" (UID: \"f3799555-a5eb-44d8-99c8-6df387241cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"
Nov 02 17:45:00.374790 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.374696    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dfpxb\" (UniqueName: \"kubernetes.io/projected/f60eecce-a69b-4055-8c35-270689f1cdbd-kube-api-access-dfpxb\") pod \"maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r\" (UID: \"f60eecce-a69b-4055-8c35-270689f1cdbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"
Nov 02 17:45:00.375135 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.375111    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60eecce-a69b-4055-8c35-270689f1cdbd-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r\" (UID: \"f60eecce-a69b-4055-8c35-270689f1cdbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"
Nov 02 17:45:00.375594 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.375570    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3799555-a5eb-44d8-99c8-6df387241cf5-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp\" (UID: \"f3799555-a5eb-44d8-99c8-6df387241cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"
Nov 02 17:45:00.375732 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.375710    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24287229-12c3-402a-a20a-ad35eaa2ce51-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l\" (UID: \"24287229-12c3-402a-a20a-ad35eaa2ce51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"
Nov 02 17:45:00.400070 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.400024    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dfpxb\" (UniqueName: \"kubernetes.io/projected/f60eecce-a69b-4055-8c35-270689f1cdbd-kube-api-access-dfpxb\") pod \"maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r\" (UID: \"f60eecce-a69b-4055-8c35-270689f1cdbd\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"
Nov 02 17:45:00.400765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.400743    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lr56b\" (UniqueName: \"kubernetes.io/projected/24287229-12c3-402a-a20a-ad35eaa2ce51-kube-api-access-lr56b\") pod \"maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l\" (UID: \"24287229-12c3-402a-a20a-ad35eaa2ce51\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"
Nov 02 17:45:00.401585 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.401565    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hzpd7\" (UniqueName: \"kubernetes.io/projected/f3799555-a5eb-44d8-99c8-6df387241cf5-kube-api-access-hzpd7\") pod \"maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp\" (UID: \"f3799555-a5eb-44d8-99c8-6df387241cf5\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"
Nov 02 17:45:00.422071 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.422031    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:45:00.422599 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.422576    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:45:00.424400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.424377    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 17:45:00.476534 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.476503    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dxwbm\" (UniqueName: \"kubernetes.io/projected/e703782b-0ae2-4e44-8a37-cbd921b3ebec-kube-api-access-dxwbm\") pod \"maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn\" (UID: \"e703782b-0ae2-4e44-8a37-cbd921b3ebec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"
Nov 02 17:45:00.476888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.476857    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e703782b-0ae2-4e44-8a37-cbd921b3ebec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn\" (UID: \"e703782b-0ae2-4e44-8a37-cbd921b3ebec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"
Nov 02 17:45:00.477457 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.477438    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e703782b-0ae2-4e44-8a37-cbd921b3ebec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn\" (UID: \"e703782b-0ae2-4e44-8a37-cbd921b3ebec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"
Nov 02 17:45:00.505173 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.505148    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dxwbm\" (UniqueName: \"kubernetes.io/projected/e703782b-0ae2-4e44-8a37-cbd921b3ebec-kube-api-access-dxwbm\") pod \"maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn\" (UID: \"e703782b-0ae2-4e44-8a37-cbd921b3ebec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"
Nov 02 17:45:00.554347 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.554315    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"
Nov 02 17:45:00.607198 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.606939    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"
Nov 02 17:45:00.614048 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.611681    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"
Nov 02 17:45:00.634426 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.634162    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"
Nov 02 17:45:00.932309 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.932229    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"]
Nov 02 17:45:00.997929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:00.997900    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"]
Nov 02 17:45:01.035097 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:01.035068    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"]
Nov 02 17:45:01.082671 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:45:01.082638    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf60eecce_a69b_4055_8c35_270689f1cdbd.slice/crio-5f2e560a4cd96140b852ff1bf3ec3d25e8b4e00ba2990dd0e7a5d958853d5022 WatchSource:0}: Error finding container 5f2e560a4cd96140b852ff1bf3ec3d25e8b4e00ba2990dd0e7a5d958853d5022: Status 404 returned error can't find the container with id 5f2e560a4cd96140b852ff1bf3ec3d25e8b4e00ba2990dd0e7a5d958853d5022
Nov 02 17:45:01.083944 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:45:01.083914    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod24287229_12c3_402a_a20a_ad35eaa2ce51.slice/crio-54d3fad96538cc26962d92980f7896de0e4d0bd8c4d180b921eb0789a84a8606 WatchSource:0}: Error finding container 54d3fad96538cc26962d92980f7896de0e4d0bd8c4d180b921eb0789a84a8606: Status 404 returned error can't find the container with id 54d3fad96538cc26962d92980f7896de0e4d0bd8c4d180b921eb0789a84a8606
Nov 02 17:45:01.093420 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:01.093395    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"]
Nov 02 17:45:01.098250 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:45:01.098229    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode703782b_0ae2_4e44_8a37_cbd921b3ebec.slice/crio-8547261712da85e68a79d26686bdd7a1ad0ad24bc23a4479399fa1fea2d04779 WatchSource:0}: Error finding container 8547261712da85e68a79d26686bdd7a1ad0ad24bc23a4479399fa1fea2d04779: Status 404 returned error can't find the container with id 8547261712da85e68a79d26686bdd7a1ad0ad24bc23a4479399fa1fea2d04779
Nov 02 17:45:01.723229 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:01.723202    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r" event={"ID":"f60eecce-a69b-4055-8c35-270689f1cdbd","Type":"ContainerStarted","Data":"11f8df7d72f1824af24f32c7fce0be08a0d2ff28c66a768f943023dcfb0e2262"}
Nov 02 17:45:01.723601 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:01.723261    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r" event={"ID":"f60eecce-a69b-4055-8c35-270689f1cdbd","Type":"ContainerStarted","Data":"5f2e560a4cd96140b852ff1bf3ec3d25e8b4e00ba2990dd0e7a5d958853d5022"}
Nov 02 17:45:01.725002 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:01.724972    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l" event={"ID":"24287229-12c3-402a-a20a-ad35eaa2ce51","Type":"ContainerStarted","Data":"602e76d28d96a8d1c16f14e2fe4025733d7a97cff92bb7ea852fb27411f2c280"}
Nov 02 17:45:01.725540 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:01.725027    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l" event={"ID":"24287229-12c3-402a-a20a-ad35eaa2ce51","Type":"ContainerStarted","Data":"54d3fad96538cc26962d92980f7896de0e4d0bd8c4d180b921eb0789a84a8606"}
Nov 02 17:45:01.726612 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:01.726589    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn" event={"ID":"e703782b-0ae2-4e44-8a37-cbd921b3ebec","Type":"ContainerStarted","Data":"e6ac7102dcb4abb48bacc13f374a4ec5795e9bada5259d9a86b969b06a842f2c"}
Nov 02 17:45:01.726708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:01.726627    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn" event={"ID":"e703782b-0ae2-4e44-8a37-cbd921b3ebec","Type":"ContainerStarted","Data":"8547261712da85e68a79d26686bdd7a1ad0ad24bc23a4479399fa1fea2d04779"}
Nov 02 17:45:01.728264 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:01.728233    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp" event={"ID":"f3799555-a5eb-44d8-99c8-6df387241cf5","Type":"ContainerStarted","Data":"063077b59873986c66da24cf21fcdb61c31302d98b2fcf46a606ebeaacbef335"}
Nov 02 17:45:01.728365 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:01.728275    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp" event={"ID":"f3799555-a5eb-44d8-99c8-6df387241cf5","Type":"ContainerStarted","Data":"41d030e7e6da728b8d52d32d6909c235ead7e2736165b5a7a0a91a1236835315"}
Nov 02 17:45:01.767885 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:01.767784    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r" podStartSLOduration=1.767770938 podStartE2EDuration="1.767770938s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:01.764126972 +0000 UTC m=+918.255190133" watchObservedRunningTime="2026-11-02 17:45:01.767770938 +0000 UTC m=+918.258834098"
Nov 02 17:45:01.809527 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:01.809468    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp" podStartSLOduration=1.809448247 podStartE2EDuration="1.809448247s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:01.806911311 +0000 UTC m=+918.297974490" watchObservedRunningTime="2026-11-02 17:45:01.809448247 +0000 UTC m=+918.300511408"
Nov 02 17:45:01.851384 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:01.851328    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l" podStartSLOduration=1.851315416 podStartE2EDuration="1.851315416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:01.847609078 +0000 UTC m=+918.338672239" watchObservedRunningTime="2026-11-02 17:45:01.851315416 +0000 UTC m=+918.342378577"
Nov 02 17:45:01.860456 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:01.860425    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:01.889496 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:01.889449    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn" podStartSLOduration=1.889422396 podStartE2EDuration="1.889422396s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:01.888328214 +0000 UTC m=+918.379391393" watchObservedRunningTime="2026-11-02 17:45:01.889422396 +0000 UTC m=+918.380485557"
Nov 02 17:45:02.751043 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:02.751002    4456 generic.go:356] "Generic (PLEG): container finished" podID="24287229-12c3-402a-a20a-ad35eaa2ce51" containerID="602e76d28d96a8d1c16f14e2fe4025733d7a97cff92bb7ea852fb27411f2c280" exitCode=60
Nov 02 17:45:02.751440 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:02.751090    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l" event={"ID":"24287229-12c3-402a-a20a-ad35eaa2ce51","Type":"ContainerDied","Data":"602e76d28d96a8d1c16f14e2fe4025733d7a97cff92bb7ea852fb27411f2c280"}
Nov 02 17:45:02.751440 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:02.751358    4456 scope.go:122] "RemoveContainer" containerID="602e76d28d96a8d1c16f14e2fe4025733d7a97cff92bb7ea852fb27411f2c280"
Nov 02 17:45:02.753237 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:02.753216    4456 generic.go:356] "Generic (PLEG): container finished" podID="e703782b-0ae2-4e44-8a37-cbd921b3ebec" containerID="e6ac7102dcb4abb48bacc13f374a4ec5795e9bada5259d9a86b969b06a842f2c" exitCode=60
Nov 02 17:45:02.753328 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:02.753297    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn" event={"ID":"e703782b-0ae2-4e44-8a37-cbd921b3ebec","Type":"ContainerDied","Data":"e6ac7102dcb4abb48bacc13f374a4ec5795e9bada5259d9a86b969b06a842f2c"}
Nov 02 17:45:02.753557 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:02.753539    4456 scope.go:122] "RemoveContainer" containerID="e6ac7102dcb4abb48bacc13f374a4ec5795e9bada5259d9a86b969b06a842f2c"
Nov 02 17:45:02.755269 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:02.755246    4456 generic.go:356] "Generic (PLEG): container finished" podID="f3799555-a5eb-44d8-99c8-6df387241cf5" containerID="063077b59873986c66da24cf21fcdb61c31302d98b2fcf46a606ebeaacbef335" exitCode=60
Nov 02 17:45:02.755353 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:02.755337    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp" event={"ID":"f3799555-a5eb-44d8-99c8-6df387241cf5","Type":"ContainerDied","Data":"063077b59873986c66da24cf21fcdb61c31302d98b2fcf46a606ebeaacbef335"}
Nov 02 17:45:02.755543 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:02.755524    4456 scope.go:122] "RemoveContainer" containerID="063077b59873986c66da24cf21fcdb61c31302d98b2fcf46a606ebeaacbef335"
Nov 02 17:45:02.757182 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:02.757158    4456 generic.go:356] "Generic (PLEG): container finished" podID="f60eecce-a69b-4055-8c35-270689f1cdbd" containerID="11f8df7d72f1824af24f32c7fce0be08a0d2ff28c66a768f943023dcfb0e2262" exitCode=60
Nov 02 17:45:02.757295 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:02.757230    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r" event={"ID":"f60eecce-a69b-4055-8c35-270689f1cdbd","Type":"ContainerDied","Data":"11f8df7d72f1824af24f32c7fce0be08a0d2ff28c66a768f943023dcfb0e2262"}
Nov 02 17:45:02.757411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:02.757397    4456 scope.go:122] "RemoveContainer" containerID="11f8df7d72f1824af24f32c7fce0be08a0d2ff28c66a768f943023dcfb0e2262"
Nov 02 17:45:03.766767 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:03.766725    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:03.773494 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:03.773449    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn" event={"ID":"e703782b-0ae2-4e44-8a37-cbd921b3ebec","Type":"ContainerStarted","Data":"be2540d8bf6408af9b1786ced221d60a00e85be524cfafcc8648a93d2d90f961"}
Nov 02 17:45:03.776611 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:03.776582    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp" event={"ID":"f3799555-a5eb-44d8-99c8-6df387241cf5","Type":"ContainerStarted","Data":"971f47afb3f0347c254be4640183d96c63b9dbc58fddcdf7cc58d542ea570d34"}
Nov 02 17:45:03.786422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:03.785405    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r" event={"ID":"f60eecce-a69b-4055-8c35-270689f1cdbd","Type":"ContainerStarted","Data":"b786e95b725f52aee8164ffe99b04456323b09f1191f5f52d48bd3ac7218af7d"}
Nov 02 17:45:03.799791 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:03.795439    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l" event={"ID":"24287229-12c3-402a-a20a-ad35eaa2ce51","Type":"ContainerStarted","Data":"86db6bf23c87a1d16dbaabe4f8a4d13138336f092691a536cde78f5141d9a63e"}
Nov 02 17:45:04.806922 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.806876    4456 generic.go:356] "Generic (PLEG): container finished" podID="f60eecce-a69b-4055-8c35-270689f1cdbd" containerID="b786e95b725f52aee8164ffe99b04456323b09f1191f5f52d48bd3ac7218af7d" exitCode=60
Nov 02 17:45:04.807415 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.806961    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r" event={"ID":"f60eecce-a69b-4055-8c35-270689f1cdbd","Type":"ContainerDied","Data":"b786e95b725f52aee8164ffe99b04456323b09f1191f5f52d48bd3ac7218af7d"}
Nov 02 17:45:04.807415 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.807038    4456 scope.go:122] "RemoveContainer" containerID="11f8df7d72f1824af24f32c7fce0be08a0d2ff28c66a768f943023dcfb0e2262"
Nov 02 17:45:04.807647 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.807493    4456 scope.go:122] "RemoveContainer" containerID="b786e95b725f52aee8164ffe99b04456323b09f1191f5f52d48bd3ac7218af7d"
Nov 02 17:45:04.807980 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:45:04.807956    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r_odh-ai-gateway-infra(f60eecce-a69b-4055-8c35-270689f1cdbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r" podUID="f60eecce-a69b-4055-8c35-270689f1cdbd"
Nov 02 17:45:04.809834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.809793    4456 generic.go:356] "Generic (PLEG): container finished" podID="24287229-12c3-402a-a20a-ad35eaa2ce51" containerID="86db6bf23c87a1d16dbaabe4f8a4d13138336f092691a536cde78f5141d9a63e" exitCode=60
Nov 02 17:45:04.810019 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.809908    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l" event={"ID":"24287229-12c3-402a-a20a-ad35eaa2ce51","Type":"ContainerDied","Data":"86db6bf23c87a1d16dbaabe4f8a4d13138336f092691a536cde78f5141d9a63e"}
Nov 02 17:45:04.810301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.810271    4456 scope.go:122] "RemoveContainer" containerID="86db6bf23c87a1d16dbaabe4f8a4d13138336f092691a536cde78f5141d9a63e"
Nov 02 17:45:04.810647 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:45:04.810619    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l_odh-ai-gateway-infra(24287229-12c3-402a-a20a-ad35eaa2ce51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l" podUID="24287229-12c3-402a-a20a-ad35eaa2ce51"
Nov 02 17:45:04.812696 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.812666    4456 generic.go:356] "Generic (PLEG): container finished" podID="e703782b-0ae2-4e44-8a37-cbd921b3ebec" containerID="be2540d8bf6408af9b1786ced221d60a00e85be524cfafcc8648a93d2d90f961" exitCode=60
Nov 02 17:45:04.812789 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.812752    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn" event={"ID":"e703782b-0ae2-4e44-8a37-cbd921b3ebec","Type":"ContainerDied","Data":"be2540d8bf6408af9b1786ced221d60a00e85be524cfafcc8648a93d2d90f961"}
Nov 02 17:45:04.813287 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.813260    4456 scope.go:122] "RemoveContainer" containerID="be2540d8bf6408af9b1786ced221d60a00e85be524cfafcc8648a93d2d90f961"
Nov 02 17:45:04.813681 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:45:04.813652    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn_odh-ai-gateway-infra(e703782b-0ae2-4e44-8a37-cbd921b3ebec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn" podUID="e703782b-0ae2-4e44-8a37-cbd921b3ebec"
Nov 02 17:45:04.821841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.819231    4456 generic.go:356] "Generic (PLEG): container finished" podID="f3799555-a5eb-44d8-99c8-6df387241cf5" containerID="971f47afb3f0347c254be4640183d96c63b9dbc58fddcdf7cc58d542ea570d34" exitCode=60
Nov 02 17:45:04.821841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.819441    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp" event={"ID":"f3799555-a5eb-44d8-99c8-6df387241cf5","Type":"ContainerDied","Data":"971f47afb3f0347c254be4640183d96c63b9dbc58fddcdf7cc58d542ea570d34"}
Nov 02 17:45:04.821841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.819633    4456 scope.go:122] "RemoveContainer" containerID="971f47afb3f0347c254be4640183d96c63b9dbc58fddcdf7cc58d542ea570d34"
Nov 02 17:45:04.821841 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:45:04.819949    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp_odh-ai-gateway-infra(f3799555-a5eb-44d8-99c8-6df387241cf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp" podUID="f3799555-a5eb-44d8-99c8-6df387241cf5"
Nov 02 17:45:04.824513 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.824494    4456 scope.go:122] "RemoveContainer" containerID="602e76d28d96a8d1c16f14e2fe4025733d7a97cff92bb7ea852fb27411f2c280"
Nov 02 17:45:04.848971 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.848944    4456 scope.go:122] "RemoveContainer" containerID="e6ac7102dcb4abb48bacc13f374a4ec5795e9bada5259d9a86b969b06a842f2c"
Nov 02 17:45:04.865050 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:04.865029    4456 scope.go:122] "RemoveContainer" containerID="063077b59873986c66da24cf21fcdb61c31302d98b2fcf46a606ebeaacbef335"
Nov 02 17:45:05.726241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:05.726201    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:05.830004 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:05.829967    4456 scope.go:122] "RemoveContainer" containerID="86db6bf23c87a1d16dbaabe4f8a4d13138336f092691a536cde78f5141d9a63e"
Nov 02 17:45:05.830448 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:45:05.830421    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l_odh-ai-gateway-infra(24287229-12c3-402a-a20a-ad35eaa2ce51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l" podUID="24287229-12c3-402a-a20a-ad35eaa2ce51"
Nov 02 17:45:05.831834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:05.831797    4456 scope.go:122] "RemoveContainer" containerID="be2540d8bf6408af9b1786ced221d60a00e85be524cfafcc8648a93d2d90f961"
Nov 02 17:45:05.832168 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:45:05.832146    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn_odh-ai-gateway-infra(e703782b-0ae2-4e44-8a37-cbd921b3ebec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn" podUID="e703782b-0ae2-4e44-8a37-cbd921b3ebec"
Nov 02 17:45:05.833637 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:05.833615    4456 scope.go:122] "RemoveContainer" containerID="971f47afb3f0347c254be4640183d96c63b9dbc58fddcdf7cc58d542ea570d34"
Nov 02 17:45:05.833959 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:45:05.833932    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp_odh-ai-gateway-infra(f3799555-a5eb-44d8-99c8-6df387241cf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp" podUID="f3799555-a5eb-44d8-99c8-6df387241cf5"
Nov 02 17:45:05.835362 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:05.835344    4456 scope.go:122] "RemoveContainer" containerID="b786e95b725f52aee8164ffe99b04456323b09f1191f5f52d48bd3ac7218af7d"
Nov 02 17:45:05.835619 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:45:05.835601    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r_odh-ai-gateway-infra(f60eecce-a69b-4055-8c35-270689f1cdbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r" podUID="f60eecce-a69b-4055-8c35-270689f1cdbd"
Nov 02 17:45:06.903492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:06.903454    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:07.344749 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.344712    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"]
Nov 02 17:45:07.350921 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.350895    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.362144 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.362117    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-workload-socket\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.362273 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.362172    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-token\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.362362 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.362310    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fxnr7\" (UniqueName: \"kubernetes.io/projected/954cad70-ab73-4391-b7f4-4fc0d0d23405-kube-api-access-fxnr7\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.362422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.362365    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-workload-certs\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.362422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.362399    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-envoy\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.362530 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.362494    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954cad70-ab73-4391-b7f4-4fc0d0d23405-istiod-ca-cert\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.362788 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.362553    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-data\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.362788 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.362606    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-credential-socket\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.362788 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.362680    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-podinfo\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.378165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.378135    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"]
Nov 02 17:45:07.387857 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.385894    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-223215e4-openshift-default-dockercfg-6zl6w\""
Nov 02 17:45:07.447272 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.447232    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:07.464463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.464432    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-podinfo\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.464594 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.464560    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-workload-socket\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.464671 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.464616    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-token\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.464716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.464675    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fxnr7\" (UniqueName: \"kubernetes.io/projected/954cad70-ab73-4391-b7f4-4fc0d0d23405-kube-api-access-fxnr7\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.464768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.464717    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-workload-certs\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.464768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.464749    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-envoy\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.465012 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.464866    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954cad70-ab73-4391-b7f4-4fc0d0d23405-istiod-ca-cert\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.465012 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.464931    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-data\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.465012 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.464985    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-credential-socket\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.465172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.465130    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-workload-certs\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.465409 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.465385    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-credential-socket\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.465514 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.465422    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-workload-socket\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.465514 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.465483    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-data\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.465986 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.465960    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954cad70-ab73-4391-b7f4-4fc0d0d23405-istiod-ca-cert\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.467308 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.467287    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-envoy\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.467610 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.467591    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-podinfo\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.493224 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.493201    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-token\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.496122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.496091    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fxnr7\" (UniqueName: \"kubernetes.io/projected/954cad70-ab73-4391-b7f4-4fc0d0d23405-kube-api-access-fxnr7\") pod \"e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") " pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:07.671897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.667132    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:07.683841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:07.681519    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:08.153512 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:45:08.153469    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod954cad70_ab73_4391_b7f4_4fc0d0d23405.slice/crio-f4cb6fc250aeae88b962ad3e135857381e70782479712e42763f92672a49bf60 WatchSource:0}: Error finding container f4cb6fc250aeae88b962ad3e135857381e70782479712e42763f92672a49bf60: Status 404 returned error can't find the container with id f4cb6fc250aeae88b962ad3e135857381e70782479712e42763f92672a49bf60
Nov 02 17:45:08.154497 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:08.154467    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"]
Nov 02 17:45:08.156470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:08.156431    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:08.156576 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:08.156515    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:08.156576 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:08.156553    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:08.195987 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:08.195957    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:08.352961 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:08.352924    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:08.699444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:08.697299    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:08.930838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:08.921891    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl" event={"ID":"954cad70-ab73-4391-b7f4-4fc0d0d23405","Type":"ContainerStarted","Data":"5ca00e66cd5fa095e01b2632ebb64bbed1a2c1cb5c657074b3e485ddd2505856"}
Nov 02 17:45:08.930838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:08.921995    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl" event={"ID":"954cad70-ab73-4391-b7f4-4fc0d0d23405","Type":"ContainerStarted","Data":"f4cb6fc250aeae88b962ad3e135857381e70782479712e42763f92672a49bf60"}
Nov 02 17:45:08.930838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:08.923190    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:09.457484 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:09.457448    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:09.634424 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:09.634395    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:09.684270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:09.684065    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:09.709897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:09.692387    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:09.760870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:09.756716    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl" podStartSLOduration=2.756694418 podStartE2EDuration="2.756694418s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:08.978942519 +0000 UTC m=+925.470005705" watchObservedRunningTime="2026-11-02 17:45:09.756694418 +0000 UTC m=+926.247757580"
Nov 02 17:45:09.801841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:09.801184    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:09.933743 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:09.933709    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:09.934991 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:09.934968    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:45:10.683765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:10.683712    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:10.849661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:10.849629    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:11.752865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:11.752796    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:12.462666 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:12.462630    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:12.632771 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:12.632735    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:12.817039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:12.814657    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"]
Nov 02 17:45:12.889281 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:12.889231    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:12.919223 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:12.919186    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"
Nov 02 17:45:12.922177 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:12.922147    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"]
Nov 02 17:45:12.942679 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:12.942645    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:45:13.073282 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:13.073175    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:13.108160 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:13.108116    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d861846c-2815-4acf-8d17-ef905e132572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-671ad5c9-gjvpj\" (UID: \"d861846c-2815-4acf-8d17-ef905e132572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"
Nov 02 17:45:13.108381 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:13.108363    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pv2fb\" (UniqueName: \"kubernetes.io/projected/d861846c-2815-4acf-8d17-ef905e132572-kube-api-access-pv2fb\") pod \"maas-api-revoke-sub-671ad5c9-gjvpj\" (UID: \"d861846c-2815-4acf-8d17-ef905e132572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"
Nov 02 17:45:13.108486 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:13.108471    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d861846c-2815-4acf-8d17-ef905e132572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-671ad5c9-gjvpj\" (UID: \"d861846c-2815-4acf-8d17-ef905e132572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"
Nov 02 17:45:13.209773 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:13.209742    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pv2fb\" (UniqueName: \"kubernetes.io/projected/d861846c-2815-4acf-8d17-ef905e132572-kube-api-access-pv2fb\") pod \"maas-api-revoke-sub-671ad5c9-gjvpj\" (UID: \"d861846c-2815-4acf-8d17-ef905e132572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"
Nov 02 17:45:13.209977 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:13.209839    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d861846c-2815-4acf-8d17-ef905e132572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-671ad5c9-gjvpj\" (UID: \"d861846c-2815-4acf-8d17-ef905e132572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"
Nov 02 17:45:13.209977 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:13.209954    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d861846c-2815-4acf-8d17-ef905e132572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-671ad5c9-gjvpj\" (UID: \"d861846c-2815-4acf-8d17-ef905e132572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"
Nov 02 17:45:13.210730 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:13.210664    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d861846c-2815-4acf-8d17-ef905e132572-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-671ad5c9-gjvpj\" (UID: \"d861846c-2815-4acf-8d17-ef905e132572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"
Nov 02 17:45:13.212656 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:13.212629    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d861846c-2815-4acf-8d17-ef905e132572-maas-api-xks-ca\") pod \"maas-api-revoke-sub-671ad5c9-gjvpj\" (UID: \"d861846c-2815-4acf-8d17-ef905e132572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"
Nov 02 17:45:13.234698 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:13.234671    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pv2fb\" (UniqueName: \"kubernetes.io/projected/d861846c-2815-4acf-8d17-ef905e132572-kube-api-access-pv2fb\") pod \"maas-api-revoke-sub-671ad5c9-gjvpj\" (UID: \"d861846c-2815-4acf-8d17-ef905e132572\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"
Nov 02 17:45:13.246481 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:13.246459    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"
Nov 02 17:45:13.579546 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:13.579518    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"]
Nov 02 17:45:13.581233 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:45:13.581177    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd861846c_2815_4acf_8d17_ef905e132572.slice/crio-f0c43f541d260d0773beba36648e6efb01233518c47058bec95d267872431fa6 WatchSource:0}: Error finding container f0c43f541d260d0773beba36648e6efb01233518c47058bec95d267872431fa6: Status 404 returned error can't find the container with id f0c43f541d260d0773beba36648e6efb01233518c47058bec95d267872431fa6
Nov 02 17:45:13.968707 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:13.968673    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj" event={"ID":"d861846c-2815-4acf-8d17-ef905e132572","Type":"ContainerStarted","Data":"f0c43f541d260d0773beba36648e6efb01233518c47058bec95d267872431fa6"}
Nov 02 17:45:13.981439 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:13.981406    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:14.609656 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:14.608232    4456 scope.go:122] "RemoveContainer" containerID="be2540d8bf6408af9b1786ced221d60a00e85be524cfafcc8648a93d2d90f961"
Nov 02 17:45:14.727974 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:14.727921    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:14.906703 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:14.906615    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:15.192232 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:15.192189    4456 generic.go:356] "Generic (PLEG): container finished" podID="d861846c-2815-4acf-8d17-ef905e132572" containerID="55281d76e5a115ec6a71d8e85c50b4bf336cfd1d51bc2e59b260acf04fb60adb" exitCode=0
Nov 02 17:45:15.192618 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:15.192238    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj" event={"ID":"d861846c-2815-4acf-8d17-ef905e132572","Type":"ContainerDied","Data":"55281d76e5a115ec6a71d8e85c50b4bf336cfd1d51bc2e59b260acf04fb60adb"}
Nov 02 17:45:15.608032 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:15.607989    4456 scope.go:122] "RemoveContainer" containerID="971f47afb3f0347c254be4640183d96c63b9dbc58fddcdf7cc58d542ea570d34"
Nov 02 17:45:16.080471 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.080413    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:16.166851 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:45:16.166741    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:45:16.208536 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.208504    4456 generic.go:356] "Generic (PLEG): container finished" podID="e703782b-0ae2-4e44-8a37-cbd921b3ebec" containerID="57a4e183674da4e161a23758c3417332cb2339a292e4b17341f08eceed094db4" exitCode=60
Nov 02 17:45:16.209043 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.208565    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn" event={"ID":"e703782b-0ae2-4e44-8a37-cbd921b3ebec","Type":"ContainerDied","Data":"57a4e183674da4e161a23758c3417332cb2339a292e4b17341f08eceed094db4"}
Nov 02 17:45:16.209043 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.208655    4456 scope.go:122] "RemoveContainer" containerID="be2540d8bf6408af9b1786ced221d60a00e85be524cfafcc8648a93d2d90f961"
Nov 02 17:45:16.209262 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.209244    4456 scope.go:122] "RemoveContainer" containerID="57a4e183674da4e161a23758c3417332cb2339a292e4b17341f08eceed094db4"
Nov 02 17:45:16.209683 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:45:16.209658    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn_odh-ai-gateway-infra(e703782b-0ae2-4e44-8a37-cbd921b3ebec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn" podUID="e703782b-0ae2-4e44-8a37-cbd921b3ebec"
Nov 02 17:45:16.213440 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.213405    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp" event={"ID":"f3799555-a5eb-44d8-99c8-6df387241cf5","Type":"ContainerStarted","Data":"e023e263493a7a2b20846a79f7345930dfa5e0a29ba83e326d32e7ca0d2f2d74"}
Nov 02 17:45:16.247964 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.247898    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:16.413060 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.408090    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:16.582649 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.582619    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"
Nov 02 17:45:16.607159 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.607126    4456 scope.go:122] "RemoveContainer" containerID="86db6bf23c87a1d16dbaabe4f8a4d13138336f092691a536cde78f5141d9a63e"
Nov 02 17:45:16.607363 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.607343    4456 scope.go:122] "RemoveContainer" containerID="b786e95b725f52aee8164ffe99b04456323b09f1191f5f52d48bd3ac7218af7d"
Nov 02 17:45:16.723929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.723897    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d861846c-2815-4acf-8d17-ef905e132572-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d861846c-2815-4acf-8d17-ef905e132572-maas-api-openshift-ca\") pod \"d861846c-2815-4acf-8d17-ef905e132572\" (UID: \"d861846c-2815-4acf-8d17-ef905e132572\") "
Nov 02 17:45:16.724099 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.724039    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d861846c-2815-4acf-8d17-ef905e132572-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d861846c-2815-4acf-8d17-ef905e132572-maas-api-xks-ca\") pod \"d861846c-2815-4acf-8d17-ef905e132572\" (UID: \"d861846c-2815-4acf-8d17-ef905e132572\") "
Nov 02 17:45:16.724197 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.724150    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d861846c-2815-4acf-8d17-ef905e132572-kube-api-access-pv2fb\" (UniqueName: \"kubernetes.io/projected/d861846c-2815-4acf-8d17-ef905e132572-kube-api-access-pv2fb\") pod \"d861846c-2815-4acf-8d17-ef905e132572\" (UID: \"d861846c-2815-4acf-8d17-ef905e132572\") "
Nov 02 17:45:16.724364 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.724339    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d861846c-2815-4acf-8d17-ef905e132572-maas-api-openshift-ca" pod "d861846c-2815-4acf-8d17-ef905e132572" (UID: "d861846c-2815-4acf-8d17-ef905e132572"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:16.725470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.725436    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d861846c-2815-4acf-8d17-ef905e132572-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:45:16.726668 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.726637    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d861846c-2815-4acf-8d17-ef905e132572-maas-api-xks-ca" pod "d861846c-2815-4acf-8d17-ef905e132572" (UID: "d861846c-2815-4acf-8d17-ef905e132572"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:45:16.726774 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.726672    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d861846c-2815-4acf-8d17-ef905e132572-kube-api-access-pv2fb" pod "d861846c-2815-4acf-8d17-ef905e132572" (UID: "d861846c-2815-4acf-8d17-ef905e132572"). InnerVolumeSpecName "kube-api-access-pv2fb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:16.836838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.827014    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d861846c-2815-4acf-8d17-ef905e132572-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:45:16.836838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:16.827051    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pv2fb\" (UniqueName: \"kubernetes.io/projected/d861846c-2815-4acf-8d17-ef905e132572-kube-api-access-pv2fb\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:45:17.224793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.224756    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj" event={"ID":"d861846c-2815-4acf-8d17-ef905e132572","Type":"ContainerDied","Data":"f0c43f541d260d0773beba36648e6efb01233518c47058bec95d267872431fa6"}
Nov 02 17:45:17.224793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.224831    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f0c43f541d260d0773beba36648e6efb01233518c47058bec95d267872431fa6"
Nov 02 17:45:17.224793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.224774    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"
Nov 02 17:45:17.227390 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.227363    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l" event={"ID":"24287229-12c3-402a-a20a-ad35eaa2ce51","Type":"ContainerStarted","Data":"df220ca820f632d49019eb368db2340fe791bd179ae20c2b836907cf4adf7e21"}
Nov 02 17:45:17.231492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.231463    4456 generic.go:356] "Generic (PLEG): container finished" podID="f3799555-a5eb-44d8-99c8-6df387241cf5" containerID="e023e263493a7a2b20846a79f7345930dfa5e0a29ba83e326d32e7ca0d2f2d74" exitCode=60
Nov 02 17:45:17.231602 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.231549    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp" event={"ID":"f3799555-a5eb-44d8-99c8-6df387241cf5","Type":"ContainerDied","Data":"e023e263493a7a2b20846a79f7345930dfa5e0a29ba83e326d32e7ca0d2f2d74"}
Nov 02 17:45:17.231653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.231602    4456 scope.go:122] "RemoveContainer" containerID="971f47afb3f0347c254be4640183d96c63b9dbc58fddcdf7cc58d542ea570d34"
Nov 02 17:45:17.232120 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.232103    4456 scope.go:122] "RemoveContainer" containerID="e023e263493a7a2b20846a79f7345930dfa5e0a29ba83e326d32e7ca0d2f2d74"
Nov 02 17:45:17.232503 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:45:17.232480    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp_odh-ai-gateway-infra(f3799555-a5eb-44d8-99c8-6df387241cf5)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp" podUID="f3799555-a5eb-44d8-99c8-6df387241cf5"
Nov 02 17:45:17.233790 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.233771    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r" event={"ID":"f60eecce-a69b-4055-8c35-270689f1cdbd","Type":"ContainerStarted","Data":"24b80bc1b9a71413e8366f3e348f7d59e641676914c692cdb87b3aaea9a1d432"}
Nov 02 17:45:17.274062 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.274032    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"]
Nov 02 17:45:17.325805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.325772    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"]
Nov 02 17:45:17.482708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.482677    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"
Nov 02 17:45:17.559858 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.554423    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e703782b-0ae2-4e44-8a37-cbd921b3ebec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e703782b-0ae2-4e44-8a37-cbd921b3ebec-maas-api-service-ca\") pod \"e703782b-0ae2-4e44-8a37-cbd921b3ebec\" (UID: \"e703782b-0ae2-4e44-8a37-cbd921b3ebec\") "
Nov 02 17:45:17.559858 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.554708    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e703782b-0ae2-4e44-8a37-cbd921b3ebec-kube-api-access-dxwbm\" (UniqueName: \"kubernetes.io/projected/e703782b-0ae2-4e44-8a37-cbd921b3ebec-kube-api-access-dxwbm\") pod \"e703782b-0ae2-4e44-8a37-cbd921b3ebec\" (UID: \"e703782b-0ae2-4e44-8a37-cbd921b3ebec\") "
Nov 02 17:45:17.559858 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.555681    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e703782b-0ae2-4e44-8a37-cbd921b3ebec-maas-api-service-ca" pod "e703782b-0ae2-4e44-8a37-cbd921b3ebec" (UID: "e703782b-0ae2-4e44-8a37-cbd921b3ebec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:17.573793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.571195    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"]
Nov 02 17:45:17.573793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.572045    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e703782b-0ae2-4e44-8a37-cbd921b3ebec" containerName="cleanup"
Nov 02 17:45:17.573793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.572062    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e703782b-0ae2-4e44-8a37-cbd921b3ebec" containerName="cleanup"
Nov 02 17:45:17.573793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.572072    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d861846c-2815-4acf-8d17-ef905e132572" containerName="revoke-keys"
Nov 02 17:45:17.573793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.572077    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d861846c-2815-4acf-8d17-ef905e132572" containerName="revoke-keys"
Nov 02 17:45:17.573793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.572112    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e703782b-0ae2-4e44-8a37-cbd921b3ebec" containerName="cleanup"
Nov 02 17:45:17.573793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.572116    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e703782b-0ae2-4e44-8a37-cbd921b3ebec" containerName="cleanup"
Nov 02 17:45:17.573793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.572209    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e703782b-0ae2-4e44-8a37-cbd921b3ebec" containerName="cleanup"
Nov 02 17:45:17.573793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.572217    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e703782b-0ae2-4e44-8a37-cbd921b3ebec" containerName="cleanup"
Nov 02 17:45:17.573793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.572228    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d861846c-2815-4acf-8d17-ef905e132572" containerName="revoke-keys"
Nov 02 17:45:17.576188 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.576158    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.578225 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.578204    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e703782b-0ae2-4e44-8a37-cbd921b3ebec-kube-api-access-dxwbm" pod "e703782b-0ae2-4e44-8a37-cbd921b3ebec" (UID: "e703782b-0ae2-4e44-8a37-cbd921b3ebec"). InnerVolumeSpecName "kube-api-access-dxwbm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:17.613209 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.613182    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-ait-30a972ba-openshift-default-dockercfg-zcw59\""
Nov 02 17:45:17.637803 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.637778    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"]
Nov 02 17:45:17.656831 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.656784    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-workload-certs\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.656975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.656894    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-envoy\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.656975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.656926    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-workload-socket\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.657089 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.656979    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-podinfo\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.657089 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.657026    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5dx67\" (UniqueName: \"kubernetes.io/projected/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-kube-api-access-5dx67\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.657197 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.657162    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istiod-ca-cert\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.657270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.657252    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-credential-socket\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.657380 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.657366    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-token\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.657432 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.657395    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-data\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.657532 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.657518    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/e703782b-0ae2-4e44-8a37-cbd921b3ebec-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:45:17.657583 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.657539    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dxwbm\" (UniqueName: \"kubernetes.io/projected/e703782b-0ae2-4e44-8a37-cbd921b3ebec-kube-api-access-dxwbm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:45:17.692472 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.692441    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:17.759856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.759631    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-credential-socket\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.759856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.759739    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-token\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.759856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.759767    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-data\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.761388 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.761335    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-credential-socket\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.762472 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.761845    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-workload-certs\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.762472 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.761942    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-envoy\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.762472 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.761975    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-workload-socket\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.762472 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.762002    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-data\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.762472 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.762043    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-podinfo\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.762472 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.762102    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5dx67\" (UniqueName: \"kubernetes.io/projected/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-kube-api-access-5dx67\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.762472 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.762240    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istiod-ca-cert\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.763044 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.763019    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istiod-ca-cert\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.763529 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.763256    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-workload-socket\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.763529 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.763493    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-workload-certs\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.770143 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.770113    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-envoy\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.796435 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.796404    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-podinfo\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.811609 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.811384    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-token\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.812186 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.812161    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5dx67\" (UniqueName: \"kubernetes.io/projected/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-kube-api-access-5dx67\") pod \"e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") " pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:17.860268 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.860072    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:17.919420 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:17.919189    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:18.245677 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.245651    4456 generic.go:356] "Generic (PLEG): container finished" podID="24287229-12c3-402a-a20a-ad35eaa2ce51" containerID="df220ca820f632d49019eb368db2340fe791bd179ae20c2b836907cf4adf7e21" exitCode=60
Nov 02 17:45:18.246080 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.245737    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l" event={"ID":"24287229-12c3-402a-a20a-ad35eaa2ce51","Type":"ContainerDied","Data":"df220ca820f632d49019eb368db2340fe791bd179ae20c2b836907cf4adf7e21"}
Nov 02 17:45:18.246080 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.245802    4456 scope.go:122] "RemoveContainer" containerID="86db6bf23c87a1d16dbaabe4f8a4d13138336f092691a536cde78f5141d9a63e"
Nov 02 17:45:18.246310 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.246276    4456 scope.go:122] "RemoveContainer" containerID="df220ca820f632d49019eb368db2340fe791bd179ae20c2b836907cf4adf7e21"
Nov 02 17:45:18.246690 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:45:18.246672    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l_odh-ai-gateway-infra(24287229-12c3-402a-a20a-ad35eaa2ce51)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l" podUID="24287229-12c3-402a-a20a-ad35eaa2ce51"
Nov 02 17:45:18.248354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.248334    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"
Nov 02 17:45:18.248354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.248342    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn" event={"ID":"e703782b-0ae2-4e44-8a37-cbd921b3ebec","Type":"ContainerDied","Data":"8547261712da85e68a79d26686bdd7a1ad0ad24bc23a4479399fa1fea2d04779"}
Nov 02 17:45:18.251417 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:45:18.250793    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfa32838f_7553_4cf6_adbc_aa7bdada9c0f.slice/crio-9f4522c874b7fa537a3f69697667033adbe6f59d88b9aadc1bab11bd7e6af39d WatchSource:0}: Error finding container 9f4522c874b7fa537a3f69697667033adbe6f59d88b9aadc1bab11bd7e6af39d: Status 404 returned error can't find the container with id 9f4522c874b7fa537a3f69697667033adbe6f59d88b9aadc1bab11bd7e6af39d
Nov 02 17:45:18.252496 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.252168    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"]
Nov 02 17:45:18.255083 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.254411    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:18.255083 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.254491    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:18.255083 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.254539    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:18.258308 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.258274    4456 generic.go:356] "Generic (PLEG): container finished" podID="f60eecce-a69b-4055-8c35-270689f1cdbd" containerID="24b80bc1b9a71413e8366f3e348f7d59e641676914c692cdb87b3aaea9a1d432" exitCode=60
Nov 02 17:45:18.258356 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.258341    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r" event={"ID":"f60eecce-a69b-4055-8c35-270689f1cdbd","Type":"ContainerDied","Data":"24b80bc1b9a71413e8366f3e348f7d59e641676914c692cdb87b3aaea9a1d432"}
Nov 02 17:45:18.258742 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.258725    4456 scope.go:122] "RemoveContainer" containerID="24b80bc1b9a71413e8366f3e348f7d59e641676914c692cdb87b3aaea9a1d432"
Nov 02 17:45:18.259003 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:45:18.258984    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r_odh-ai-gateway-infra(f60eecce-a69b-4055-8c35-270689f1cdbd)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r" podUID="f60eecce-a69b-4055-8c35-270689f1cdbd"
Nov 02 17:45:18.268297 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.268279    4456 scope.go:122] "RemoveContainer" containerID="57a4e183674da4e161a23758c3417332cb2339a292e4b17341f08eceed094db4"
Nov 02 17:45:18.280580 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.280470    4456 scope.go:122] "RemoveContainer" containerID="b786e95b725f52aee8164ffe99b04456323b09f1191f5f52d48bd3ac7218af7d"
Nov 02 17:45:18.310851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.310788    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"]
Nov 02 17:45:18.351873 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.351835    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"]
Nov 02 17:45:18.357980 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.357959    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894025-k2kzn"]
Nov 02 17:45:18.448188 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.448161    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"
Nov 02 17:45:18.473005 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.467891    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"]
Nov 02 17:45:18.502994 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.495865    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:18.504020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.503742    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f3799555-a5eb-44d8-99c8-6df387241cf5-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3799555-a5eb-44d8-99c8-6df387241cf5-maas-api-service-ca\") pod \"f3799555-a5eb-44d8-99c8-6df387241cf5\" (UID: \"f3799555-a5eb-44d8-99c8-6df387241cf5\") "
Nov 02 17:45:18.504020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.503915    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f3799555-a5eb-44d8-99c8-6df387241cf5-kube-api-access-hzpd7\" (UniqueName: \"kubernetes.io/projected/f3799555-a5eb-44d8-99c8-6df387241cf5-kube-api-access-hzpd7\") pod \"f3799555-a5eb-44d8-99c8-6df387241cf5\" (UID: \"f3799555-a5eb-44d8-99c8-6df387241cf5\") "
Nov 02 17:45:18.509788 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.508578    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f3799555-a5eb-44d8-99c8-6df387241cf5-kube-api-access-hzpd7" pod "f3799555-a5eb-44d8-99c8-6df387241cf5" (UID: "f3799555-a5eb-44d8-99c8-6df387241cf5"). InnerVolumeSpecName "kube-api-access-hzpd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:18.509788 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.509034    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f3799555-a5eb-44d8-99c8-6df387241cf5-maas-api-service-ca" pod "f3799555-a5eb-44d8-99c8-6df387241cf5" (UID: "f3799555-a5eb-44d8-99c8-6df387241cf5"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:18.608029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.606907    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f3799555-a5eb-44d8-99c8-6df387241cf5-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:45:18.608029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:18.606935    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hzpd7\" (UniqueName: \"kubernetes.io/projected/f3799555-a5eb-44d8-99c8-6df387241cf5-kube-api-access-hzpd7\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:45:19.282538 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:19.281121    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp" event={"ID":"f3799555-a5eb-44d8-99c8-6df387241cf5","Type":"ContainerDied","Data":"41d030e7e6da728b8d52d32d6909c235ead7e2736165b5a7a0a91a1236835315"}
Nov 02 17:45:19.282538 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:19.281187    4456 scope.go:122] "RemoveContainer" containerID="e023e263493a7a2b20846a79f7345930dfa5e0a29ba83e326d32e7ca0d2f2d74"
Nov 02 17:45:19.282538 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:19.281312    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"
Nov 02 17:45:19.287491 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:19.286398    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk" event={"ID":"fa32838f-7553-4cf6-adbc-aa7bdada9c0f","Type":"ContainerStarted","Data":"9459610172c097c21f19e4e84aaf8e60c529796cbce40d873a856cb38b13e69c"}
Nov 02 17:45:19.287491 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:19.286450    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk" event={"ID":"fa32838f-7553-4cf6-adbc-aa7bdada9c0f","Type":"ContainerStarted","Data":"9f4522c874b7fa537a3f69697667033adbe6f59d88b9aadc1bab11bd7e6af39d"}
Nov 02 17:45:19.416397 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:19.416316    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk" podStartSLOduration=2.4162943 podStartE2EDuration="2.4162943s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:19.413322106 +0000 UTC m=+935.904385315" watchObservedRunningTime="2026-11-02 17:45:19.4162943 +0000 UTC m=+935.907357462"
Nov 02 17:45:19.771449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:19.771409    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e703782b-0ae2-4e44-8a37-cbd921b3ebec" path="/var/lib/kubelet/pods/e703782b-0ae2-4e44-8a37-cbd921b3ebec/volumes"
Nov 02 17:45:19.791504 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:19.791468    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"]
Nov 02 17:45:19.815853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:19.805322    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894025-8j8hp"]
Nov 02 17:45:19.947055 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:19.947012    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:19.967173 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:19.967141    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:20.282843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:20.277944    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:20.326972 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:20.326919    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:20.366049 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:20.366017    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:20.388841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:20.385376    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:45:20.390064 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:20.390029    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 17:45:20.414991 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:20.393983    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:20.415457 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:20.394020    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:20.416159 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:20.404719    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:20.478799 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:20.477892    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:20.495966 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:20.483903    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:20.639951 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:20.629613    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:20.639951 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:20.629886    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 17:45:21.381050 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:21.345392    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:45:21.381050 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:21.348115    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:45:21.551929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:21.546848    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:21.570840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:21.564966    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:21.578847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:21.574037    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Nov 02 17:45:21.601318 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:21.597951    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 17:45:21.617026 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:21.612059    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 17:45:21.617026 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:21.612366    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:45:21.617026 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:21.613656    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 17:45:21.617026 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:21.613710    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:45:21.850208 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:21.804677    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 17:45:22.080512 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:22.080411    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:45:22.478181 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:22.478130    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:22.497862 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:22.479223    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:22.757910 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:22.684593    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 17:45:22.766849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:22.762861    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:45:22.807123 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:22.807063    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:22.853046 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:22.853001    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:22.883843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:22.861945    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:22.919544 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:22.861989    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:45:22.919544 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:22.884995    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:22.972061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:22.972011    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 17:45:22.980142 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:22.980085    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:23.222055 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.222005    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded"
Nov 02 17:45:23.493163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.493077    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f3799555-a5eb-44d8-99c8-6df387241cf5" path="/var/lib/kubelet/pods/f3799555-a5eb-44d8-99c8-6df387241cf5/volumes"
Nov 02 17:45:23.515841 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:45:23.494510    4456 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.414s"
Nov 02 17:45:23.516702 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.516674    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:45:23.516953 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.516939    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:45:23.517072 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.517062    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:45:23.517205 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.517194    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:45:23.517316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.517307    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:45:23.517429 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.517420    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:45:23.517533 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.517524    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:45:23.542756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.502060    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:23.544159 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.544119    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:23.721016 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.720968    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:23.782425 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.782331    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:23.922869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.919375    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:23.943968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.935800    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:45:23.943968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.937118    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 17:45:23.967371 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.967311    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 17:45:23.992348 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:23.974299    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:24.033934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.024781    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"
Nov 02 17:45:24.033934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.026792    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"
Nov 02 17:45:24.035547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.035503    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 17:45:24.060469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.060417    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:24.193872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.175050    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/24287229-12c3-402a-a20a-ad35eaa2ce51-kube-api-access-lr56b\" (UniqueName: \"kubernetes.io/projected/24287229-12c3-402a-a20a-ad35eaa2ce51-kube-api-access-lr56b\") pod \"24287229-12c3-402a-a20a-ad35eaa2ce51\" (UID: \"24287229-12c3-402a-a20a-ad35eaa2ce51\") "
Nov 02 17:45:24.265872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.213157    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/24287229-12c3-402a-a20a-ad35eaa2ce51-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24287229-12c3-402a-a20a-ad35eaa2ce51-maas-api-service-ca\") pod \"24287229-12c3-402a-a20a-ad35eaa2ce51\" (UID: \"24287229-12c3-402a-a20a-ad35eaa2ce51\") "
Nov 02 17:45:24.265872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.213323    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f60eecce-a69b-4055-8c35-270689f1cdbd-kube-api-access-dfpxb\" (UniqueName: \"kubernetes.io/projected/f60eecce-a69b-4055-8c35-270689f1cdbd-kube-api-access-dfpxb\") pod \"f60eecce-a69b-4055-8c35-270689f1cdbd\" (UID: \"f60eecce-a69b-4055-8c35-270689f1cdbd\") "
Nov 02 17:45:24.265872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.213546    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f60eecce-a69b-4055-8c35-270689f1cdbd-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60eecce-a69b-4055-8c35-270689f1cdbd-maas-api-service-ca\") pod \"f60eecce-a69b-4055-8c35-270689f1cdbd\" (UID: \"f60eecce-a69b-4055-8c35-270689f1cdbd\") "
Nov 02 17:45:24.265872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.221920    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f60eecce-a69b-4055-8c35-270689f1cdbd-maas-api-service-ca" pod "f60eecce-a69b-4055-8c35-270689f1cdbd" (UID: "f60eecce-a69b-4055-8c35-270689f1cdbd"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:24.330894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.324782    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f60eecce-a69b-4055-8c35-270689f1cdbd-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:45:24.361372 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.361336    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/24287229-12c3-402a-a20a-ad35eaa2ce51-maas-api-service-ca" pod "24287229-12c3-402a-a20a-ad35eaa2ce51" (UID: "24287229-12c3-402a-a20a-ad35eaa2ce51"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:24.480639 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.480598    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:24.541707 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.540364    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/24287229-12c3-402a-a20a-ad35eaa2ce51-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:45:24.938646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.553921    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:45:24.938646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.613864    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f60eecce-a69b-4055-8c35-270689f1cdbd-kube-api-access-dfpxb" pod "f60eecce-a69b-4055-8c35-270689f1cdbd" (UID: "f60eecce-a69b-4055-8c35-270689f1cdbd"). InnerVolumeSpecName "kube-api-access-dfpxb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:24.938646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.613982    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/24287229-12c3-402a-a20a-ad35eaa2ce51-kube-api-access-lr56b" pod "24287229-12c3-402a-a20a-ad35eaa2ce51" (UID: "24287229-12c3-402a-a20a-ad35eaa2ce51"). InnerVolumeSpecName "kube-api-access-lr56b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:24.938646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.650750    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr56b\" (UniqueName: \"kubernetes.io/projected/24287229-12c3-402a-a20a-ad35eaa2ce51-kube-api-access-lr56b\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:45:24.938646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.650784    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dfpxb\" (UniqueName: \"kubernetes.io/projected/f60eecce-a69b-4055-8c35-270689f1cdbd-kube-api-access-dfpxb\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:45:24.938646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.726981    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 17:45:24.938646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.797738    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:25.004843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.971344    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:25.004843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:24.971615    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 17:45:25.006390 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:25.006351    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:25.006707 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:25.006670    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:45:25.607750 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:25.009521    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:25.608612 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:25.037946    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:25.608913 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:25.393341    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"]
Nov 02 17:45:25.837895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:25.837858    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"
Nov 02 17:45:25.861310 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:25.839456    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f60eecce-a69b-4055-8c35-270689f1cdbd" containerName="cleanup"
Nov 02 17:45:25.896563 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:25.896530    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f60eecce-a69b-4055-8c35-270689f1cdbd" containerName="cleanup"
Nov 02 17:45:26.053691 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:25.924149    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f3799555-a5eb-44d8-99c8-6df387241cf5" containerName="cleanup"
Nov 02 17:45:26.066610 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.066576    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3799555-a5eb-44d8-99c8-6df387241cf5" containerName="cleanup"
Nov 02 17:45:26.066792 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.066780    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e703782b-0ae2-4e44-8a37-cbd921b3ebec" containerName="cleanup"
Nov 02 17:45:26.066926 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.066914    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e703782b-0ae2-4e44-8a37-cbd921b3ebec" containerName="cleanup"
Nov 02 17:45:26.067059 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.067050    4456 cpu_manager.go:414] "containerMap: removing container" podUID="24287229-12c3-402a-a20a-ad35eaa2ce51" containerName="cleanup"
Nov 02 17:45:26.067146 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.067135    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24287229-12c3-402a-a20a-ad35eaa2ce51" containerName="cleanup"
Nov 02 17:45:26.067233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.067223    4456 cpu_manager.go:414] "containerMap: removing container" podUID="24287229-12c3-402a-a20a-ad35eaa2ce51" containerName="cleanup"
Nov 02 17:45:26.067314 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.067304    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24287229-12c3-402a-a20a-ad35eaa2ce51" containerName="cleanup"
Nov 02 17:45:26.073405 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.072040    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f60eecce-a69b-4055-8c35-270689f1cdbd" containerName="cleanup"
Nov 02 17:45:26.073668 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.073645    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f60eecce-a69b-4055-8c35-270689f1cdbd" containerName="cleanup"
Nov 02 17:45:26.073776 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.073765    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f3799555-a5eb-44d8-99c8-6df387241cf5" containerName="cleanup"
Nov 02 17:45:26.073895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.073883    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3799555-a5eb-44d8-99c8-6df387241cf5" containerName="cleanup"
Nov 02 17:45:26.074244 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.074224    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f60eecce-a69b-4055-8c35-270689f1cdbd" containerName="cleanup"
Nov 02 17:45:26.074364 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.074354    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="24287229-12c3-402a-a20a-ad35eaa2ce51" containerName="cleanup"
Nov 02 17:45:26.074447 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.074439    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3799555-a5eb-44d8-99c8-6df387241cf5" containerName="cleanup"
Nov 02 17:45:26.074517 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.074508    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f60eecce-a69b-4055-8c35-270689f1cdbd" containerName="cleanup"
Nov 02 17:45:26.074583 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.074575    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3799555-a5eb-44d8-99c8-6df387241cf5" containerName="cleanup"
Nov 02 17:45:26.074658 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.074650    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f3799555-a5eb-44d8-99c8-6df387241cf5" containerName="cleanup"
Nov 02 17:45:26.074729 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.074721    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e703782b-0ae2-4e44-8a37-cbd921b3ebec" containerName="cleanup"
Nov 02 17:45:26.074809 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.074801    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="24287229-12c3-402a-a20a-ad35eaa2ce51" containerName="cleanup"
Nov 02 17:45:26.154891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:25.860350    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:26.154891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:25.870132    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:45:26.154891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.122130    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96" podUID="3891e06c-0c53-421c-b379-c38efca65905" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.67:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:26.154891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:25.986289    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"
Nov 02 17:45:26.321836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.302858    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l" event={"ID":"24287229-12c3-402a-a20a-ad35eaa2ce51","Type":"ContainerDied","Data":"54d3fad96538cc26962d92980f7896de0e4d0bd8c4d180b921eb0789a84a8606"}
Nov 02 17:45:26.321836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.309338    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"
Nov 02 17:45:26.334501 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.333912    4456 scope.go:122] "RemoveContainer" containerID="df220ca820f632d49019eb368db2340fe791bd179ae20c2b836907cf4adf7e21"
Nov 02 17:45:26.383130 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.377011    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:45:26.395593 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.394639    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:26.395593 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.394724    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:26.395593 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.394982    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:26.395593 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.395075    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:26.395593 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.395171    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:26.413850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.413744    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:45:26.449836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.440921    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:45:26.514256 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.514225    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/736a6511-6750-49a8-8830-05c9dccd495c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53683142-bmtlf\" (UID: \"736a6511-6750-49a8-8830-05c9dccd495c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"
Nov 02 17:45:26.514412 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.514332    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vxfgn\" (UniqueName: \"kubernetes.io/projected/736a6511-6750-49a8-8830-05c9dccd495c-kube-api-access-vxfgn\") pod \"maas-api-revoke-sub-53683142-bmtlf\" (UID: \"736a6511-6750-49a8-8830-05c9dccd495c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"
Nov 02 17:45:26.514585 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.514565    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/736a6511-6750-49a8-8830-05c9dccd495c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53683142-bmtlf\" (UID: \"736a6511-6750-49a8-8830-05c9dccd495c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"
Nov 02 17:45:26.519637 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.519607    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:26.539050 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.539020    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:45:26.539242 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.539081    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r" event={"ID":"f60eecce-a69b-4055-8c35-270689f1cdbd","Type":"ContainerDied","Data":"5f2e560a4cd96140b852ff1bf3ec3d25e8b4e00ba2990dd0e7a5d958853d5022"}
Nov 02 17:45:26.539242 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.539115    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"]
Nov 02 17:45:26.558743 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.539200    4456 scope.go:122] "RemoveContainer" containerID="24b80bc1b9a71413e8366f3e348f7d59e641676914c692cdb87b3aaea9a1d432"
Nov 02 17:45:26.599003 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.596064    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:45:26.599003 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.596172    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:45:26.624837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.620729    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:45:26.647253 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.647229    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:45:26.647450 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.647438    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:45:26.648736 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.648718    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:45:26.651179 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.651154    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 17:45:26.665085 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.665013    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"]
Nov 02 17:45:26.676634 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.676584    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/736a6511-6750-49a8-8830-05c9dccd495c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53683142-bmtlf\" (UID: \"736a6511-6750-49a8-8830-05c9dccd495c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"
Nov 02 17:45:26.677019 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.676999    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vxfgn\" (UniqueName: \"kubernetes.io/projected/736a6511-6750-49a8-8830-05c9dccd495c-kube-api-access-vxfgn\") pod \"maas-api-revoke-sub-53683142-bmtlf\" (UID: \"736a6511-6750-49a8-8830-05c9dccd495c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"
Nov 02 17:45:26.677449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.677429    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/736a6511-6750-49a8-8830-05c9dccd495c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53683142-bmtlf\" (UID: \"736a6511-6750-49a8-8830-05c9dccd495c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"
Nov 02 17:45:26.694875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.681506    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/736a6511-6750-49a8-8830-05c9dccd495c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53683142-bmtlf\" (UID: \"736a6511-6750-49a8-8830-05c9dccd495c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"
Nov 02 17:45:26.694875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.684552    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w6-7df526-29894025-h6b4l"]
Nov 02 17:45:26.694875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.688286    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:26.694875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.688504    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:45:26.694875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.690978    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"]
Nov 02 17:45:26.703489 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.703466    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w4-2899ab-29894025-bwc4r"]
Nov 02 17:45:26.712724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.709770    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vxfgn\" (UniqueName: \"kubernetes.io/projected/736a6511-6750-49a8-8830-05c9dccd495c-kube-api-access-vxfgn\") pod \"maas-api-revoke-sub-53683142-bmtlf\" (UID: \"736a6511-6750-49a8-8830-05c9dccd495c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"
Nov 02 17:45:26.712724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.710317    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/736a6511-6750-49a8-8830-05c9dccd495c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53683142-bmtlf\" (UID: \"736a6511-6750-49a8-8830-05c9dccd495c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"
Nov 02 17:45:26.990430 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:26.990394    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"
Nov 02 17:45:27.111510 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:27.111474    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:27.277163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:27.277084    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:27.381216 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:27.380341    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"]
Nov 02 17:45:27.624911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:27.621002    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="24287229-12c3-402a-a20a-ad35eaa2ce51" path="/var/lib/kubelet/pods/24287229-12c3-402a-a20a-ad35eaa2ce51/volumes"
Nov 02 17:45:27.624911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:27.622547    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f60eecce-a69b-4055-8c35-270689f1cdbd" path="/var/lib/kubelet/pods/f60eecce-a69b-4055-8c35-270689f1cdbd/volumes"
Nov 02 17:45:28.021122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:28.021088    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf" event={"ID":"736a6511-6750-49a8-8830-05c9dccd495c","Type":"ContainerStarted","Data":"97657c79213205657317432093eb88e1146e42258a218a154be8df535ad485e9"}
Nov 02 17:45:28.021314 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:28.021151    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf" event={"ID":"736a6511-6750-49a8-8830-05c9dccd495c","Type":"ContainerStarted","Data":"a097b6ea6d89c337e6eb02dec264ffb332095551c94d1ae6bc593f644714e6ef"}
Nov 02 17:45:28.069979 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:28.069923    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf" podStartSLOduration=4.069907563 podStartE2EDuration="4.069907563s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:28.063682589 +0000 UTC m=+944.554745749" watchObservedRunningTime="2026-11-02 17:45:28.069907563 +0000 UTC m=+944.560970723"
Nov 02 17:45:28.314074 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:28.313991    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:28.486947 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:28.486912    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:28.647926 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:28.641563    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:29.034212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.034168    4456 generic.go:356] "Generic (PLEG): container finished" podID="736a6511-6750-49a8-8830-05c9dccd495c" containerID="97657c79213205657317432093eb88e1146e42258a218a154be8df535ad485e9" exitCode=0
Nov 02 17:45:29.034399 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.034254    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf" event={"ID":"736a6511-6750-49a8-8830-05c9dccd495c","Type":"ContainerDied","Data":"97657c79213205657317432093eb88e1146e42258a218a154be8df535ad485e9"}
Nov 02 17:45:29.314249 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.314119    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"]
Nov 02 17:45:29.314957 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.314933    4456 cpu_manager.go:414] "containerMap: removing container" podUID="24287229-12c3-402a-a20a-ad35eaa2ce51" containerName="cleanup"
Nov 02 17:45:29.314957 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.314957    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="24287229-12c3-402a-a20a-ad35eaa2ce51" containerName="cleanup"
Nov 02 17:45:29.315120 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.314997    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f3799555-a5eb-44d8-99c8-6df387241cf5" containerName="cleanup"
Nov 02 17:45:29.315120 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.315005    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f3799555-a5eb-44d8-99c8-6df387241cf5" containerName="cleanup"
Nov 02 17:45:29.315120 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.315026    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f60eecce-a69b-4055-8c35-270689f1cdbd" containerName="cleanup"
Nov 02 17:45:29.315120 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.315034    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f60eecce-a69b-4055-8c35-270689f1cdbd" containerName="cleanup"
Nov 02 17:45:29.315305 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.315236    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="24287229-12c3-402a-a20a-ad35eaa2ce51" containerName="cleanup"
Nov 02 17:45:29.315305 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.315257    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f60eecce-a69b-4055-8c35-270689f1cdbd" containerName="cleanup"
Nov 02 17:45:29.319445 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.319422    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.326701 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.326672    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"]
Nov 02 17:45:29.336733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.336629    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 17:45:29.336733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.336653    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 17:45:29.336733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.336629    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 17:45:29.336954 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.336797    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dklz6\""
Nov 02 17:45:29.429702 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.429666    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.429888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.429867    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.429982 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.429933    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.430056 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.430032    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.430188 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.430171    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.430255 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.430214    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.430255 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.430242    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sfntx\" (UniqueName: \"kubernetes.io/projected/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-kube-api-access-sfntx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.531565 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.531524    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.531725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.531647    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.531725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.531687    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.531870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.531850    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.531870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.531869    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.531984 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.531883    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sfntx\" (UniqueName: \"kubernetes.io/projected/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-kube-api-access-sfntx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.531984 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.531943    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.532753 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.532723    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.534165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.534136    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.534307 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.534288    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.534463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.534441    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.534519 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.534452    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.555406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.555384    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.555942 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.555922    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sfntx\" (UniqueName: \"kubernetes.io/projected/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-kube-api-access-sfntx\") pod \"istiod-openshift-gateway-7ddd9cfdc8-tdx82\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.640164 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.640092    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:29.969691 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:45:29.969664    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0f2089f4_8ec7_45ce_a2df_0bf024b4158f.slice/crio-e8c411f43f20cb4752201faa744662d6d835f0af36906442b13f55878b99bce8 WatchSource:0}: Error finding container e8c411f43f20cb4752201faa744662d6d835f0af36906442b13f55878b99bce8: Status 404 returned error can't find the container with id e8c411f43f20cb4752201faa744662d6d835f0af36906442b13f55878b99bce8
Nov 02 17:45:29.970405 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.970382    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"]
Nov 02 17:45:29.972297 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:29.972280    4456 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:45:30.044517 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:30.044479    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" event={"ID":"0f2089f4-8ec7-45ce-a2df-0bf024b4158f","Type":"ContainerStarted","Data":"e8c411f43f20cb4752201faa744662d6d835f0af36906442b13f55878b99bce8"}
Nov 02 17:45:30.245632 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:30.245605    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"
Nov 02 17:45:30.344486 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:30.344424    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/736a6511-6750-49a8-8830-05c9dccd495c-kube-api-access-vxfgn\" (UniqueName: \"kubernetes.io/projected/736a6511-6750-49a8-8830-05c9dccd495c-kube-api-access-vxfgn\") pod \"736a6511-6750-49a8-8830-05c9dccd495c\" (UID: \"736a6511-6750-49a8-8830-05c9dccd495c\") "
Nov 02 17:45:30.344903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:30.344877    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/736a6511-6750-49a8-8830-05c9dccd495c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/736a6511-6750-49a8-8830-05c9dccd495c-maas-api-xks-ca\") pod \"736a6511-6750-49a8-8830-05c9dccd495c\" (UID: \"736a6511-6750-49a8-8830-05c9dccd495c\") "
Nov 02 17:45:30.345358 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:30.345334    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/736a6511-6750-49a8-8830-05c9dccd495c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/736a6511-6750-49a8-8830-05c9dccd495c-maas-api-openshift-ca\") pod \"736a6511-6750-49a8-8830-05c9dccd495c\" (UID: \"736a6511-6750-49a8-8830-05c9dccd495c\") "
Nov 02 17:45:30.346571 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:30.346545    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/736a6511-6750-49a8-8830-05c9dccd495c-maas-api-openshift-ca" pod "736a6511-6750-49a8-8830-05c9dccd495c" (UID: "736a6511-6750-49a8-8830-05c9dccd495c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:45:30.350527 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:30.350504    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/736a6511-6750-49a8-8830-05c9dccd495c-maas-api-xks-ca" pod "736a6511-6750-49a8-8830-05c9dccd495c" (UID: "736a6511-6750-49a8-8830-05c9dccd495c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:45:30.350721 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:30.350692    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/736a6511-6750-49a8-8830-05c9dccd495c-kube-api-access-vxfgn" pod "736a6511-6750-49a8-8830-05c9dccd495c" (UID: "736a6511-6750-49a8-8830-05c9dccd495c"). InnerVolumeSpecName "kube-api-access-vxfgn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:45:30.416737 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:30.416700    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:30.447446 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:30.447414    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vxfgn\" (UniqueName: \"kubernetes.io/projected/736a6511-6750-49a8-8830-05c9dccd495c-kube-api-access-vxfgn\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:45:30.447446 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:30.447443    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/736a6511-6750-49a8-8830-05c9dccd495c-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:45:30.447446 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:30.447452    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/736a6511-6750-49a8-8830-05c9dccd495c-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:45:30.570674 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:30.570639    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:31.056151 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:31.056111    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"
Nov 02 17:45:31.056701 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:31.056146    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf" event={"ID":"736a6511-6750-49a8-8830-05c9dccd495c","Type":"ContainerDied","Data":"a097b6ea6d89c337e6eb02dec264ffb332095551c94d1ae6bc593f644714e6ef"}
Nov 02 17:45:31.056701 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:31.056210    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a097b6ea6d89c337e6eb02dec264ffb332095551c94d1ae6bc593f644714e6ef"
Nov 02 17:45:32.471301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:32.471237    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:33.061127 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:33.061094    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"]
Nov 02 17:45:33.073350 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:33.073317    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e5dd46c7-cs5pr"]
Nov 02 17:45:33.170567 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:33.170531    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:33.622205 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:33.622170    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3f7e4b49-d0a8-4409-899b-7fd9b9e51d24" path="/var/lib/kubelet/pods/3f7e4b49-d0a8-4409-899b-7fd9b9e51d24/volumes"
Nov 02 17:45:33.981283 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:33.981233    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:34.219563 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:34.219526    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:34.374611 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:34.374515    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:34.616892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:34.616861    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:34.617005 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:34.616931    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:45:35.106725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:35.106680    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" event={"ID":"0f2089f4-8ec7-45ce-a2df-0bf024b4158f","Type":"ContainerStarted","Data":"593b4317838a7e992136f4e9e542b750536a8fd59cf1ad26c128305c332da9be"}
Nov 02 17:45:35.107243 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:35.106993    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:35.108956 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:35.108924    4456 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-tdx82 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:45:35.109085 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:35.108992    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" podUID="0f2089f4-8ec7-45ce-a2df-0bf024b4158f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:45:35.183627 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:35.183560    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" podStartSLOduration=1.5404148979999999 podStartE2EDuration="6.183540979s" totalImagesPullingTime="4.643126081s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:29 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:35.165321728 +0000 UTC m=+951.656384900" watchObservedRunningTime="2026-11-02 17:45:35.183540979 +0000 UTC m=+951.674604141"
Nov 02 17:45:36.118263 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:36.118230    4456 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-tdx82 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:45:36.118655 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:36.118298    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" podUID="0f2089f4-8ec7-45ce-a2df-0bf024b4158f" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:45:36.252624 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:36.252573    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:37.765461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:37.765424    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:37.950024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:37.949990    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:38.131010 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:38.130929    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:39.118974 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:39.118942    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 17:45:39.681463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:39.681426    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:39.958993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:39.958942    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:40.931833 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:40.931786    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:41.752319 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:41.752288    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:43.475400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:43.475364    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:43.626269 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:43.626237    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:43.771081 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:43.770997    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:45.333458 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:45.333414    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:45.483034 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:45.482999    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:45.636481 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:45.636394    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:45.783290 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:45.783216    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:46.052678 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:46.052649    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"]
Nov 02 17:45:46.054047 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:46.054017    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:46.063826 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:46.063773    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2dcd3478-d9ktv"]
Nov 02 17:45:46.166602 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:45:46.166568    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:45:47.157174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:47.157135    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:47.618774 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:47.618736    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06ef6b4f-c98a-4655-8b92-6d6930eec8a1" path="/var/lib/kubelet/pods/06ef6b4f-c98a-4655-8b92-6d6930eec8a1/volumes"
Nov 02 17:45:49.066658 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:49.065092    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:49.209840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:49.209712    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:51.077554 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:51.077520    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:53.050387 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:53.049427    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:53.927950 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:53.927911    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:54.218300 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:54.218252    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:55.359407 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:55.359357    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:57.163136 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.163097    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:57.237993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.237962    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"]
Nov 02 17:45:57.238682 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.238665    4456 cpu_manager.go:414] "containerMap: removing container" podUID="736a6511-6750-49a8-8830-05c9dccd495c" containerName="revoke-keys"
Nov 02 17:45:57.238682 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.238682    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="736a6511-6750-49a8-8830-05c9dccd495c" containerName="revoke-keys"
Nov 02 17:45:57.238839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.238798    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="736a6511-6750-49a8-8830-05c9dccd495c" containerName="revoke-keys"
Nov 02 17:45:57.242948 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.242914    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"
Nov 02 17:45:57.267389 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.267341    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"]
Nov 02 17:45:57.269742 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.269716    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:45:57.269903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.269745    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:45:57.287207 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.287160    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43ae017-8e7f-4904-863a-3cab95d728ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-223215e4-sd5x7\" (UID: \"a43ae017-8e7f-4904-863a-3cab95d728ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"
Nov 02 17:45:57.287565 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.287538    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d4r7n\" (UniqueName: \"kubernetes.io/projected/a43ae017-8e7f-4904-863a-3cab95d728ff-kube-api-access-d4r7n\") pod \"maas-api-revoke-keys-e2e-ait-223215e4-sd5x7\" (UID: \"a43ae017-8e7f-4904-863a-3cab95d728ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"
Nov 02 17:45:57.287881 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.287860    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43ae017-8e7f-4904-863a-3cab95d728ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-223215e4-sd5x7\" (UID: \"a43ae017-8e7f-4904-863a-3cab95d728ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"
Nov 02 17:45:57.390291 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.390157    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43ae017-8e7f-4904-863a-3cab95d728ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-223215e4-sd5x7\" (UID: \"a43ae017-8e7f-4904-863a-3cab95d728ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"
Nov 02 17:45:57.390291 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.390245    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d4r7n\" (UniqueName: \"kubernetes.io/projected/a43ae017-8e7f-4904-863a-3cab95d728ff-kube-api-access-d4r7n\") pod \"maas-api-revoke-keys-e2e-ait-223215e4-sd5x7\" (UID: \"a43ae017-8e7f-4904-863a-3cab95d728ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"
Nov 02 17:45:57.392993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.390337    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43ae017-8e7f-4904-863a-3cab95d728ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-223215e4-sd5x7\" (UID: \"a43ae017-8e7f-4904-863a-3cab95d728ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"
Nov 02 17:45:57.392993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.391206    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43ae017-8e7f-4904-863a-3cab95d728ff-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-223215e4-sd5x7\" (UID: \"a43ae017-8e7f-4904-863a-3cab95d728ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"
Nov 02 17:45:57.395772 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.393909    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43ae017-8e7f-4904-863a-3cab95d728ff-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-223215e4-sd5x7\" (UID: \"a43ae017-8e7f-4904-863a-3cab95d728ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"
Nov 02 17:45:57.415585 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.415534    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d4r7n\" (UniqueName: \"kubernetes.io/projected/a43ae017-8e7f-4904-863a-3cab95d728ff-kube-api-access-d4r7n\") pod \"maas-api-revoke-keys-e2e-ait-223215e4-sd5x7\" (UID: \"a43ae017-8e7f-4904-863a-3cab95d728ff\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"
Nov 02 17:45:57.562751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.562717    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"
Nov 02 17:45:57.901475 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:57.901438    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"]
Nov 02 17:45:57.901653 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:45:57.901630    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda43ae017_8e7f_4904_863a_3cab95d728ff.slice/crio-99f214e7e904f973856a3442cb8dd69fec6674ced1c34f257034623546a2dd83 WatchSource:0}: Error finding container 99f214e7e904f973856a3442cb8dd69fec6674ced1c34f257034623546a2dd83: Status 404 returned error can't find the container with id 99f214e7e904f973856a3442cb8dd69fec6674ced1c34f257034623546a2dd83
Nov 02 17:45:58.641957 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:58.641849    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7" event={"ID":"a43ae017-8e7f-4904-863a-3cab95d728ff","Type":"ContainerStarted","Data":"09d2b13134cdabf9f47a6d7f1eba5e151f3dddf0d0303477f9cb35987e7f49b5"}
Nov 02 17:45:58.641957 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:58.641914    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7" event={"ID":"a43ae017-8e7f-4904-863a-3cab95d728ff","Type":"ContainerStarted","Data":"99f214e7e904f973856a3442cb8dd69fec6674ced1c34f257034623546a2dd83"}
Nov 02 17:45:58.687243 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:58.687172    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7" podStartSLOduration=1.687152023 podStartE2EDuration="1.687152023s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:45:57 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:45:58.683798323 +0000 UTC m=+975.174861479" watchObservedRunningTime="2026-11-02 17:45:58.687152023 +0000 UTC m=+975.178215186"
Nov 02 17:45:59.228599 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:59.228551    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:45:59.653747 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:59.653711    4456 generic.go:356] "Generic (PLEG): container finished" podID="a43ae017-8e7f-4904-863a-3cab95d728ff" containerID="09d2b13134cdabf9f47a6d7f1eba5e151f3dddf0d0303477f9cb35987e7f49b5" exitCode=0
Nov 02 17:45:59.654254 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:45:59.653848    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7" event={"ID":"a43ae017-8e7f-4904-863a-3cab95d728ff","Type":"ContainerDied","Data":"09d2b13134cdabf9f47a6d7f1eba5e151f3dddf0d0303477f9cb35987e7f49b5"}
Nov 02 17:46:00.893317 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:00.893278    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"
Nov 02 17:46:00.958265 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:00.958231    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a43ae017-8e7f-4904-863a-3cab95d728ff-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43ae017-8e7f-4904-863a-3cab95d728ff-maas-api-openshift-ca\") pod \"a43ae017-8e7f-4904-863a-3cab95d728ff\" (UID: \"a43ae017-8e7f-4904-863a-3cab95d728ff\") "
Nov 02 17:46:00.958471 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:00.958330    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a43ae017-8e7f-4904-863a-3cab95d728ff-kube-api-access-d4r7n\" (UniqueName: \"kubernetes.io/projected/a43ae017-8e7f-4904-863a-3cab95d728ff-kube-api-access-d4r7n\") pod \"a43ae017-8e7f-4904-863a-3cab95d728ff\" (UID: \"a43ae017-8e7f-4904-863a-3cab95d728ff\") "
Nov 02 17:46:00.958471 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:00.958462    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a43ae017-8e7f-4904-863a-3cab95d728ff-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43ae017-8e7f-4904-863a-3cab95d728ff-maas-api-xks-ca\") pod \"a43ae017-8e7f-4904-863a-3cab95d728ff\" (UID: \"a43ae017-8e7f-4904-863a-3cab95d728ff\") "
Nov 02 17:46:00.958705 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:00.958675    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a43ae017-8e7f-4904-863a-3cab95d728ff-maas-api-openshift-ca" pod "a43ae017-8e7f-4904-863a-3cab95d728ff" (UID: "a43ae017-8e7f-4904-863a-3cab95d728ff"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:46:00.959336 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:00.959309    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a43ae017-8e7f-4904-863a-3cab95d728ff-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:00.961006 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:00.960974    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a43ae017-8e7f-4904-863a-3cab95d728ff-kube-api-access-d4r7n" pod "a43ae017-8e7f-4904-863a-3cab95d728ff" (UID: "a43ae017-8e7f-4904-863a-3cab95d728ff"). InnerVolumeSpecName "kube-api-access-d4r7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:46:00.961495 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:00.961467    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a43ae017-8e7f-4904-863a-3cab95d728ff-maas-api-xks-ca" pod "a43ae017-8e7f-4904-863a-3cab95d728ff" (UID: "a43ae017-8e7f-4904-863a-3cab95d728ff"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:46:01.049095 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:01.049051    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"]
Nov 02 17:46:01.056801 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:01.056772    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-367340d0-kvcgj"]
Nov 02 17:46:01.060182 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:01.060154    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:01.060793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:01.060773    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a43ae017-8e7f-4904-863a-3cab95d728ff-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:01.060855 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:01.060798    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d4r7n\" (UniqueName: \"kubernetes.io/projected/a43ae017-8e7f-4904-863a-3cab95d728ff-kube-api-access-d4r7n\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:01.620309 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:01.620214    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3484eb28-e4be-4867-a6e6-aac7908e8124" path="/var/lib/kubelet/pods/3484eb28-e4be-4867-a6e6-aac7908e8124/volumes"
Nov 02 17:46:01.676991 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:01.676960    4456 generic.go:356] "Generic (PLEG): container finished" podID="954cad70-ab73-4391-b7f4-4fc0d0d23405" containerID="5ca00e66cd5fa095e01b2632ebb64bbed1a2c1cb5c657074b3e485ddd2505856" exitCode=0
Nov 02 17:46:01.677166 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:01.677032    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl" event={"ID":"954cad70-ab73-4391-b7f4-4fc0d0d23405","Type":"ContainerDied","Data":"5ca00e66cd5fa095e01b2632ebb64bbed1a2c1cb5c657074b3e485ddd2505856"}
Nov 02 17:46:01.677466 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:01.677443    4456 scope.go:122] "RemoveContainer" containerID="5ca00e66cd5fa095e01b2632ebb64bbed1a2c1cb5c657074b3e485ddd2505856"
Nov 02 17:46:01.678670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:01.678644    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7" event={"ID":"a43ae017-8e7f-4904-863a-3cab95d728ff","Type":"ContainerDied","Data":"99f214e7e904f973856a3442cb8dd69fec6674ced1c34f257034623546a2dd83"}
Nov 02 17:46:01.678769 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:01.678691    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"
Nov 02 17:46:01.678769 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:01.678692    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="99f214e7e904f973856a3442cb8dd69fec6674ced1c34f257034623546a2dd83"
Nov 02 17:46:01.679079 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:01.679048    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:01.679144 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:01.679128    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:01.679183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:01.679164    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:02.715708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:02.715663    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl" event={"ID":"954cad70-ab73-4391-b7f4-4fc0d0d23405","Type":"ContainerStarted","Data":"209b7f29ac51f0da039ef8bf59b223284d1728f1f08612ffb2870456ef69247b"}
Nov 02 17:46:03.043030 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:03.042701    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:03.701844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:03.692638    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:46:03.701844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:03.692894    4456 patch_prober.go:29] interesting pod/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 17:46:03.701844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:03.692945    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl" podUID="954cad70-ab73-4391-b7f4-4fc0d0d23405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 17:46:04.048425 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.048332    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"]
Nov 02 17:46:04.059454 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.059421    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f14b6302-tv6k4"]
Nov 02 17:46:04.684842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.684172    4456 patch_prober.go:29] interesting pod/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 17:46:04.684842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.684235    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl" podUID="954cad70-ab73-4391-b7f4-4fc0d0d23405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 17:46:04.877581 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.877546    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:04.950204 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.950091    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"]
Nov 02 17:46:04.951492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.951457    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a43ae017-8e7f-4904-863a-3cab95d728ff" containerName="revoke-keys"
Nov 02 17:46:04.951492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.951490    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a43ae017-8e7f-4904-863a-3cab95d728ff" containerName="revoke-keys"
Nov 02 17:46:04.951738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.951720    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a43ae017-8e7f-4904-863a-3cab95d728ff" containerName="revoke-keys"
Nov 02 17:46:04.955856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.955836    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:04.976507 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.976473    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:04.976759 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.976738    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:04.976921 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.976902    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:04.977161 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.977135    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-245hq\" (UniqueName: \"kubernetes.io/projected/07f874bb-122b-4794-ab59-736495e5cfdc-kube-api-access-245hq\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:04.977259 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.977199    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:04.977317 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.977304    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f874bb-122b-4794-ab59-736495e5cfdc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:04.977474 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.977455    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f874bb-122b-4794-ab59-736495e5cfdc-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:04.977528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.977514    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f874bb-122b-4794-ab59-736495e5cfdc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:04.977587 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.977543    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:04.978649 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:04.978623    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"]
Nov 02 17:46:05.078611 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.078583    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f874bb-122b-4794-ab59-736495e5cfdc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.079189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.079167    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f874bb-122b-4794-ab59-736495e5cfdc-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.079322 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.079307    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f874bb-122b-4794-ab59-736495e5cfdc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.079409 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.079396    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.079706 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.079687    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.083840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.079856    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.106237 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.106199    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.106518 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.106490    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-245hq\" (UniqueName: \"kubernetes.io/projected/07f874bb-122b-4794-ab59-736495e5cfdc-kube-api-access-245hq\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.106720 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.106701    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.106870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.106030    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.112693 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.086986    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f874bb-122b-4794-ab59-736495e5cfdc-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.112869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.112106    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.113827 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.113608    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.113966 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.113638    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f874bb-122b-4794-ab59-736495e5cfdc-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.114069 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.113700    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.119980 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.119910    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.131547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.131499    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f874bb-122b-4794-ab59-736495e5cfdc-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.132100 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.132077    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-245hq\" (UniqueName: \"kubernetes.io/projected/07f874bb-122b-4794-ab59-736495e5cfdc-kube-api-access-245hq\") pod \"maas-default-gateway-openshift-default-76f4876667-vrvf2\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.275534 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.275450    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:05.619941 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.619856    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a0f4e882-e05a-4ad5-8cfe-20117be35a46" path="/var/lib/kubelet/pods/a0f4e882-e05a-4ad5-8cfe-20117be35a46/volumes"
Nov 02 17:46:05.657540 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:46:05.657503    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod07f874bb_122b_4794_ab59_736495e5cfdc.slice/crio-d35cd6bf175b5a6b7c0ef33fdc1bc03a436a9ed8ef8a2f695ebb1d2d633e9c6e WatchSource:0}: Error finding container d35cd6bf175b5a6b7c0ef33fdc1bc03a436a9ed8ef8a2f695ebb1d2d633e9c6e: Status 404 returned error can't find the container with id d35cd6bf175b5a6b7c0ef33fdc1bc03a436a9ed8ef8a2f695ebb1d2d633e9c6e
Nov 02 17:46:05.659827 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.659783    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"]
Nov 02 17:46:05.660606 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.660571    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:05.660666 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.660650    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:05.660703 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.660694    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:05.703845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.694216    4456 patch_prober.go:29] interesting pod/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 17:46:05.703845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.694298    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl" podUID="954cad70-ab73-4391-b7f4-4fc0d0d23405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 17:46:05.743974 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:05.743941    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" event={"ID":"07f874bb-122b-4794-ab59-736495e5cfdc","Type":"ContainerStarted","Data":"d35cd6bf175b5a6b7c0ef33fdc1bc03a436a9ed8ef8a2f695ebb1d2d633e9c6e"}
Nov 02 17:46:06.687883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:06.686262    4456 patch_prober.go:29] interesting pod/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 17:46:06.687883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:06.686335    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl" podUID="954cad70-ab73-4391-b7f4-4fc0d0d23405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 17:46:06.825839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:06.813968    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" event={"ID":"07f874bb-122b-4794-ab59-736495e5cfdc","Type":"ContainerStarted","Data":"aff8684245e6811977d173dba51890ae99616b8073fb04935c6a2893e1dadaf3"}
Nov 02 17:46:06.897039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:06.895262    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podStartSLOduration=2.895240908 podStartE2EDuration="2.895240908s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:06.878194582 +0000 UTC m=+983.369257742" watchObservedRunningTime="2026-11-02 17:46:06.895240908 +0000 UTC m=+983.386304070"
Nov 02 17:46:06.962481 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:06.952351    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:07.276422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:07.276251    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:07.277695 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:07.277662    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vrvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 17:46:07.277871 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:07.277717    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 17:46:07.737837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:07.708003    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:07.739158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:07.719969    4456 patch_prober.go:29] interesting pod/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused" start-of-body=
Nov 02 17:46:07.739435 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:07.739397    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl" podUID="954cad70-ab73-4391-b7f4-4fc0d0d23405" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.107:15021/healthz/ready\": dial tcp 10.134.0.107:15021: connect: connection refused"
Nov 02 17:46:07.910844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:07.890870    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:08.278861 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:08.278243    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vrvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 17:46:08.278861 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:08.278314    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 17:46:08.905251 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:08.905217    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:46:08.918940 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:08.907595    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:46:08.922909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:08.922876    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:46:09.105465 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:09.105400    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:09.276089 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:09.276056    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vrvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 17:46:09.276286 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:09.276128    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 17:46:10.036740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:10.036708    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"]
Nov 02 17:46:10.048230 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:10.048195    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2250f85a-s246f"]
Nov 02 17:46:10.276551 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:10.276517    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vrvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 17:46:10.276721 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:10.276569    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 17:46:10.420609 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:10.420503    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:10.565526 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:10.565494    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:11.200294 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:11.200260    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:11.276518 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:11.276483    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vrvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 17:46:11.276686 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:11.276548    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 17:46:11.635092 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:11.635009    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="26efd847-04dd-4445-9f92-da25a98bd953" path="/var/lib/kubelet/pods/26efd847-04dd-4445-9f92-da25a98bd953/volumes"
Nov 02 17:46:12.276935 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:12.276900    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vrvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 17:46:12.277456 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:12.276956    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 17:46:12.781676 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:12.781620    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:13.278238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.278200    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vrvf2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused" start-of-body=
Nov 02 17:46:13.281041 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.278932    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": dial tcp 10.134.0.113:15021: connect: connection refused"
Nov 02 17:46:13.438530 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.438491    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:13.578936 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.578796    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"]
Nov 02 17:46:13.615170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.608771    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:13.638975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.638944    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"]
Nov 02 17:46:13.652113 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:46:13.652057    4456 status_manager.go:1045] "Failed to get status for pod" err="pods \"maas-api-revoke-keys-e2e-ait-223215e4-sd5x7\" is forbidden: User \"system:node:ip-10-0-141-87.ec2.internal\" cannot get resource \"pods\" in API group \"\" in the namespace \"odh-ai-gateway-infra\": no relationship found between node 'ip-10-0-141-87.ec2.internal' and this object" podUID="a43ae017-8e7f-4904-863a-3cab95d728ff" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-223215e4-sd5x7"
Nov 02 17:46:13.771603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.771528    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:13.783854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.781636    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"]
Nov 02 17:46:13.813100 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.813071    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"
Nov 02 17:46:13.819757 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.817556    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"]
Nov 02 17:46:13.827588 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.827562    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"]
Nov 02 17:46:13.828003 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.827967    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl" podUID="954cad70-ab73-4391-b7f4-4fc0d0d23405" containerName="istio-proxy" containerID="cri-o://209b7f29ac51f0da039ef8bf59b223284d1728f1f08612ffb2870456ef69247b" gracePeriod=30
Nov 02 17:46:13.836431 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.836203    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:46:13.837196 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.837109    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:46:13.887536 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.887490    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c465e47-a754-42da-b48d-c41acd9fbedf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk\" (UID: \"8c465e47-a754-42da-b48d-c41acd9fbedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"
Nov 02 17:46:13.888109 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.888078    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b2g2r\" (UniqueName: \"kubernetes.io/projected/8c465e47-a754-42da-b48d-c41acd9fbedf-kube-api-access-b2g2r\") pod \"maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk\" (UID: \"8c465e47-a754-42da-b48d-c41acd9fbedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"
Nov 02 17:46:13.888468 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:13.888446    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c465e47-a754-42da-b48d-c41acd9fbedf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk\" (UID: \"8c465e47-a754-42da-b48d-c41acd9fbedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"
Nov 02 17:46:14.052894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:14.052780    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:14.079889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:14.067452    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c465e47-a754-42da-b48d-c41acd9fbedf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk\" (UID: \"8c465e47-a754-42da-b48d-c41acd9fbedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"
Nov 02 17:46:14.079889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:14.067581    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c465e47-a754-42da-b48d-c41acd9fbedf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk\" (UID: \"8c465e47-a754-42da-b48d-c41acd9fbedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"
Nov 02 17:46:14.079889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:14.067869    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b2g2r\" (UniqueName: \"kubernetes.io/projected/8c465e47-a754-42da-b48d-c41acd9fbedf-kube-api-access-b2g2r\") pod \"maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk\" (UID: \"8c465e47-a754-42da-b48d-c41acd9fbedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"
Nov 02 17:46:14.079889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:14.069103    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c465e47-a754-42da-b48d-c41acd9fbedf-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk\" (UID: \"8c465e47-a754-42da-b48d-c41acd9fbedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"
Nov 02 17:46:14.079889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:14.077238    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c465e47-a754-42da-b48d-c41acd9fbedf-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk\" (UID: \"8c465e47-a754-42da-b48d-c41acd9fbedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"
Nov 02 17:46:14.106476 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:14.104331    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b2g2r\" (UniqueName: \"kubernetes.io/projected/8c465e47-a754-42da-b48d-c41acd9fbedf-kube-api-access-b2g2r\") pod \"maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk\" (UID: \"8c465e47-a754-42da-b48d-c41acd9fbedf\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"
Nov 02 17:46:14.155840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:14.152071    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"
Nov 02 17:46:14.550097 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:14.550018    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:14.551959 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:14.551928    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"]
Nov 02 17:46:14.556837 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:46:14.556782    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c465e47_a754_42da_b48d_c41acd9fbedf.slice/crio-59e7cb8940d2124d39a68c079a2d1acaf63d9eca19449875ea756a80ee856474 WatchSource:0}: Error finding container 59e7cb8940d2124d39a68c079a2d1acaf63d9eca19449875ea756a80ee856474: Status 404 returned error can't find the container with id 59e7cb8940d2124d39a68c079a2d1acaf63d9eca19449875ea756a80ee856474
Nov 02 17:46:14.717930 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:14.717877    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:14.869791 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:14.869698    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:15.108895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:15.108857    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk" event={"ID":"8c465e47-a754-42da-b48d-c41acd9fbedf","Type":"ContainerStarted","Data":"6fd13213f39a7920736e9fee8d27e3fc7530e7a22bde03f78e52c8b4a2141052"}
Nov 02 17:46:15.109080 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:15.108918    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk" event={"ID":"8c465e47-a754-42da-b48d-c41acd9fbedf","Type":"ContainerStarted","Data":"59e7cb8940d2124d39a68c079a2d1acaf63d9eca19449875ea756a80ee856474"}
Nov 02 17:46:15.156525 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:15.156414    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk" podStartSLOduration=2.156396083 podStartE2EDuration="2.156396083s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:15.154922477 +0000 UTC m=+991.645985696" watchObservedRunningTime="2026-11-02 17:46:15.156396083 +0000 UTC m=+991.647459244"
Nov 02 17:46:15.162622 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:15.162594    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:15.163064 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:15.163039    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:15.164068 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:15.164051    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 17:46:15.413005 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:15.412912    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:15.602318 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:15.602284    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:15.619866 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:15.619831    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a43ae017-8e7f-4904-863a-3cab95d728ff" path="/var/lib/kubelet/pods/a43ae017-8e7f-4904-863a-3cab95d728ff/volumes"
Nov 02 17:46:16.041529 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:16.041492    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"]
Nov 02 17:46:16.052619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:16.052587    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-aaba299d-6qjfx"]
Nov 02 17:46:16.093282 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:16.093194    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:16.209884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:16.208901    4456 generic.go:356] "Generic (PLEG): container finished" podID="8c465e47-a754-42da-b48d-c41acd9fbedf" containerID="6fd13213f39a7920736e9fee8d27e3fc7530e7a22bde03f78e52c8b4a2141052" exitCode=0
Nov 02 17:46:16.224841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:16.222134    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk" event={"ID":"8c465e47-a754-42da-b48d-c41acd9fbedf","Type":"ContainerDied","Data":"6fd13213f39a7920736e9fee8d27e3fc7530e7a22bde03f78e52c8b4a2141052"}
Nov 02 17:46:16.244758 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:16.244724    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:16.302595 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:46:16.302546    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:46:16.409097 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:16.407243    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:17.428998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:17.428968    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"
Nov 02 17:46:17.574940 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:17.574854    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c465e47-a754-42da-b48d-c41acd9fbedf-kube-api-access-b2g2r\" (UniqueName: \"kubernetes.io/projected/8c465e47-a754-42da-b48d-c41acd9fbedf-kube-api-access-b2g2r\") pod \"8c465e47-a754-42da-b48d-c41acd9fbedf\" (UID: \"8c465e47-a754-42da-b48d-c41acd9fbedf\") "
Nov 02 17:46:17.575081 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:17.575019    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c465e47-a754-42da-b48d-c41acd9fbedf-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c465e47-a754-42da-b48d-c41acd9fbedf-maas-api-openshift-ca\") pod \"8c465e47-a754-42da-b48d-c41acd9fbedf\" (UID: \"8c465e47-a754-42da-b48d-c41acd9fbedf\") "
Nov 02 17:46:17.575152 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:17.575135    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8c465e47-a754-42da-b48d-c41acd9fbedf-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c465e47-a754-42da-b48d-c41acd9fbedf-maas-api-xks-ca\") pod \"8c465e47-a754-42da-b48d-c41acd9fbedf\" (UID: \"8c465e47-a754-42da-b48d-c41acd9fbedf\") "
Nov 02 17:46:17.575435 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:17.575406    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c465e47-a754-42da-b48d-c41acd9fbedf-maas-api-openshift-ca" pod "8c465e47-a754-42da-b48d-c41acd9fbedf" (UID: "8c465e47-a754-42da-b48d-c41acd9fbedf"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:46:17.576418 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:17.576385    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8c465e47-a754-42da-b48d-c41acd9fbedf-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:17.577374 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:17.577347    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c465e47-a754-42da-b48d-c41acd9fbedf-kube-api-access-b2g2r" pod "8c465e47-a754-42da-b48d-c41acd9fbedf" (UID: "8c465e47-a754-42da-b48d-c41acd9fbedf"). InnerVolumeSpecName "kube-api-access-b2g2r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:46:17.577470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:17.577437    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8c465e47-a754-42da-b48d-c41acd9fbedf-maas-api-xks-ca" pod "8c465e47-a754-42da-b48d-c41acd9fbedf" (UID: "8c465e47-a754-42da-b48d-c41acd9fbedf"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:46:17.618137 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:17.618108    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1ab4274b-2b63-4b09-88e3-62e77c2c40a6" path="/var/lib/kubelet/pods/1ab4274b-2b63-4b09-88e3-62e77c2c40a6/volumes"
Nov 02 17:46:17.678810 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:17.678775    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b2g2r\" (UniqueName: \"kubernetes.io/projected/8c465e47-a754-42da-b48d-c41acd9fbedf-kube-api-access-b2g2r\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:17.678978 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:17.678833    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8c465e47-a754-42da-b48d-c41acd9fbedf-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:17.935219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:17.935191    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:18.256317 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:18.256232    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk" event={"ID":"8c465e47-a754-42da-b48d-c41acd9fbedf","Type":"ContainerDied","Data":"59e7cb8940d2124d39a68c079a2d1acaf63d9eca19449875ea756a80ee856474"}
Nov 02 17:46:18.256502 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:18.256478    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"
Nov 02 17:46:18.256653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:18.256639    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59e7cb8940d2124d39a68c079a2d1acaf63d9eca19449875ea756a80ee856474"
Nov 02 17:46:18.331612 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:18.331563    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:18.612917 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:18.612797    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:19.184690 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.184652    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:19.205509 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.205096    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:46:19.337418 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.337376    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/954cad70-ab73-4391-b7f4-4fc0d0d23405-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954cad70-ab73-4391-b7f4-4fc0d0d23405-istiod-ca-cert\") pod \"954cad70-ab73-4391-b7f4-4fc0d0d23405\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") "
Nov 02 17:46:19.344751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.344725    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-podinfo\") pod \"954cad70-ab73-4391-b7f4-4fc0d0d23405\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") "
Nov 02 17:46:19.369545 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.345403    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-workload-socket\") pod \"954cad70-ab73-4391-b7f4-4fc0d0d23405\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") "
Nov 02 17:46:19.395849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.395447    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-envoy\") pod \"954cad70-ab73-4391-b7f4-4fc0d0d23405\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") "
Nov 02 17:46:19.395849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.395585    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-credential-socket\") pod \"954cad70-ab73-4391-b7f4-4fc0d0d23405\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") "
Nov 02 17:46:19.395849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.395644    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-token\" (UniqueName: \"kubernetes.io/projected/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-token\") pod \"954cad70-ab73-4391-b7f4-4fc0d0d23405\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") "
Nov 02 17:46:19.395849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.395725    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-workload-certs\") pod \"954cad70-ab73-4391-b7f4-4fc0d0d23405\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") "
Nov 02 17:46:19.395849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.395835    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/954cad70-ab73-4391-b7f4-4fc0d0d23405-kube-api-access-fxnr7\" (UniqueName: \"kubernetes.io/projected/954cad70-ab73-4391-b7f4-4fc0d0d23405-kube-api-access-fxnr7\") pod \"954cad70-ab73-4391-b7f4-4fc0d0d23405\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") "
Nov 02 17:46:19.396258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.395974    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-data\") pod \"954cad70-ab73-4391-b7f4-4fc0d0d23405\" (UID: \"954cad70-ab73-4391-b7f4-4fc0d0d23405\") "
Nov 02 17:46:19.412198 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.344037    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/954cad70-ab73-4391-b7f4-4fc0d0d23405-istiod-ca-cert" pod "954cad70-ab73-4391-b7f4-4fc0d0d23405" (UID: "954cad70-ab73-4391-b7f4-4fc0d0d23405"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:46:19.412198 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.360121    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-workload-socket" pod "954cad70-ab73-4391-b7f4-4fc0d0d23405" (UID: "954cad70-ab73-4391-b7f4-4fc0d0d23405"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:46:19.412198 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.399926    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-podinfo" pod "954cad70-ab73-4391-b7f4-4fc0d0d23405" (UID: "954cad70-ab73-4391-b7f4-4fc0d0d23405"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:46:19.412198 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.400140    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-credential-socket" pod "954cad70-ab73-4391-b7f4-4fc0d0d23405" (UID: "954cad70-ab73-4391-b7f4-4fc0d0d23405"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:46:19.412198 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.400875    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-workload-certs" pod "954cad70-ab73-4391-b7f4-4fc0d0d23405" (UID: "954cad70-ab73-4391-b7f4-4fc0d0d23405"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:46:19.426635 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.426590    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-data" pod "954cad70-ab73-4391-b7f4-4fc0d0d23405" (UID: "954cad70-ab73-4391-b7f4-4fc0d0d23405"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:46:19.428258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.428228    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:19.436202 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.436178    4456 generic.go:356] "Generic (PLEG): container finished" podID="954cad70-ab73-4391-b7f4-4fc0d0d23405" containerID="209b7f29ac51f0da039ef8bf59b223284d1728f1f08612ffb2870456ef69247b" exitCode=0
Nov 02 17:46:19.436361 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.436345    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl" event={"ID":"954cad70-ab73-4391-b7f4-4fc0d0d23405","Type":"ContainerDied","Data":"209b7f29ac51f0da039ef8bf59b223284d1728f1f08612ffb2870456ef69247b"}
Nov 02 17:46:19.436487 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.436473    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl" event={"ID":"954cad70-ab73-4391-b7f4-4fc0d0d23405","Type":"ContainerDied","Data":"f4cb6fc250aeae88b962ad3e135857381e70782479712e42763f92672a49bf60"}
Nov 02 17:46:19.436582 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.436572    4456 scope.go:122] "RemoveContainer" containerID="209b7f29ac51f0da039ef8bf59b223284d1728f1f08612ffb2870456ef69247b"
Nov 02 17:46:19.436829 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.436795    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"
Nov 02 17:46:19.464768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.464268    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-envoy" pod "954cad70-ab73-4391-b7f4-4fc0d0d23405" (UID: "954cad70-ab73-4391-b7f4-4fc0d0d23405"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:46:19.464768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.464587    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954cad70-ab73-4391-b7f4-4fc0d0d23405-kube-api-access-fxnr7" pod "954cad70-ab73-4391-b7f4-4fc0d0d23405" (UID: "954cad70-ab73-4391-b7f4-4fc0d0d23405"). InnerVolumeSpecName "kube-api-access-fxnr7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:46:19.464768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.464690    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-token" pod "954cad70-ab73-4391-b7f4-4fc0d0d23405" (UID: "954cad70-ab73-4391-b7f4-4fc0d0d23405"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:46:19.485654 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.485627    4456 scope.go:122] "RemoveContainer" containerID="5ca00e66cd5fa095e01b2632ebb64bbed1a2c1cb5c657074b3e485ddd2505856"
Nov 02 17:46:19.514669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.514532    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:19.514669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.514560    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fxnr7\" (UniqueName: \"kubernetes.io/projected/954cad70-ab73-4391-b7f4-4fc0d0d23405-kube-api-access-fxnr7\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:19.514669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.514570    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:19.514669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.514580    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/954cad70-ab73-4391-b7f4-4fc0d0d23405-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:19.514669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.514588    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:19.514669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.514596    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:19.514669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.514603    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:19.514669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.514611    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/954cad70-ab73-4391-b7f4-4fc0d0d23405-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:19.514669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.514618    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/954cad70-ab73-4391-b7f4-4fc0d0d23405-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:19.518956 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.518932    4456 scope.go:122] "RemoveContainer" containerID="209b7f29ac51f0da039ef8bf59b223284d1728f1f08612ffb2870456ef69247b"
Nov 02 17:46:19.519381 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:46:19.519350    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"209b7f29ac51f0da039ef8bf59b223284d1728f1f08612ffb2870456ef69247b\": container with ID starting with 209b7f29ac51f0da039ef8bf59b223284d1728f1f08612ffb2870456ef69247b not found: ID does not exist" containerID="209b7f29ac51f0da039ef8bf59b223284d1728f1f08612ffb2870456ef69247b"
Nov 02 17:46:19.519468 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.519398    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"209b7f29ac51f0da039ef8bf59b223284d1728f1f08612ffb2870456ef69247b"} err="failed to get container status \"209b7f29ac51f0da039ef8bf59b223284d1728f1f08612ffb2870456ef69247b\": rpc error: code = NotFound desc = could not find container \"209b7f29ac51f0da039ef8bf59b223284d1728f1f08612ffb2870456ef69247b\": container with ID starting with 209b7f29ac51f0da039ef8bf59b223284d1728f1f08612ffb2870456ef69247b not found: ID does not exist"
Nov 02 17:46:19.519468 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.519423    4456 scope.go:122] "RemoveContainer" containerID="5ca00e66cd5fa095e01b2632ebb64bbed1a2c1cb5c657074b3e485ddd2505856"
Nov 02 17:46:19.519797 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:46:19.519770    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5ca00e66cd5fa095e01b2632ebb64bbed1a2c1cb5c657074b3e485ddd2505856\": container with ID starting with 5ca00e66cd5fa095e01b2632ebb64bbed1a2c1cb5c657074b3e485ddd2505856 not found: ID does not exist" containerID="5ca00e66cd5fa095e01b2632ebb64bbed1a2c1cb5c657074b3e485ddd2505856"
Nov 02 17:46:19.519889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.519806    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5ca00e66cd5fa095e01b2632ebb64bbed1a2c1cb5c657074b3e485ddd2505856"} err="failed to get container status \"5ca00e66cd5fa095e01b2632ebb64bbed1a2c1cb5c657074b3e485ddd2505856\": rpc error: code = NotFound desc = could not find container \"5ca00e66cd5fa095e01b2632ebb64bbed1a2c1cb5c657074b3e485ddd2505856\": container with ID starting with 5ca00e66cd5fa095e01b2632ebb64bbed1a2c1cb5c657074b3e485ddd2505856 not found: ID does not exist"
Nov 02 17:46:19.791233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.791145    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"]
Nov 02 17:46:19.803737 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.803701    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-223215e4-openshift-default-5984f7cdbf-4nncl"]
Nov 02 17:46:19.870905 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:19.870876    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:20.012573 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.012534    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"]
Nov 02 17:46:20.013692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.013664    4456 cpu_manager.go:414] "containerMap: removing container" podUID="954cad70-ab73-4391-b7f4-4fc0d0d23405" containerName="istio-proxy"
Nov 02 17:46:20.013692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.013690    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954cad70-ab73-4391-b7f4-4fc0d0d23405" containerName="istio-proxy"
Nov 02 17:46:20.013931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.013747    4456 cpu_manager.go:414] "containerMap: removing container" podUID="954cad70-ab73-4391-b7f4-4fc0d0d23405" containerName="istio-proxy"
Nov 02 17:46:20.013931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.013756    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="954cad70-ab73-4391-b7f4-4fc0d0d23405" containerName="istio-proxy"
Nov 02 17:46:20.013931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.013780    4456 cpu_manager.go:414] "containerMap: removing container" podUID="8c465e47-a754-42da-b48d-c41acd9fbedf" containerName="revoke-keys"
Nov 02 17:46:20.013931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.013788    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c465e47-a754-42da-b48d-c41acd9fbedf" containerName="revoke-keys"
Nov 02 17:46:20.014139 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.013983    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="954cad70-ab73-4391-b7f4-4fc0d0d23405" containerName="istio-proxy"
Nov 02 17:46:20.014139 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.014005    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="954cad70-ab73-4391-b7f4-4fc0d0d23405" containerName="istio-proxy"
Nov 02 17:46:20.014139 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.014018    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c465e47-a754-42da-b48d-c41acd9fbedf" containerName="revoke-keys"
Nov 02 17:46:20.020258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.020233    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.042090 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.041523    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"]
Nov 02 17:46:20.069862 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.069788    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:20.140751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.140711    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dggp4\" (UniqueName: \"kubernetes.io/projected/8ef79a10-292b-42d9-aab0-53bee3e43ed2-kube-api-access-dggp4\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.140976 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.140761    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef79a10-292b-42d9-aab0-53bee3e43ed2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.140976 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.140894    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef79a10-292b-42d9-aab0-53bee3e43ed2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.140976 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.140960    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef79a10-292b-42d9-aab0-53bee3e43ed2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.141140 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.141039    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef79a10-292b-42d9-aab0-53bee3e43ed2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.141202 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.141181    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef79a10-292b-42d9-aab0-53bee3e43ed2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.141299 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.141283    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef79a10-292b-42d9-aab0-53bee3e43ed2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.141352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.141329    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef79a10-292b-42d9-aab0-53bee3e43ed2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.141400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.141386    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef79a10-292b-42d9-aab0-53bee3e43ed2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.242700 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.242662    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef79a10-292b-42d9-aab0-53bee3e43ed2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.242878 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.242720    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef79a10-292b-42d9-aab0-53bee3e43ed2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.242878 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.242799    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef79a10-292b-42d9-aab0-53bee3e43ed2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.242878 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.242852    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef79a10-292b-42d9-aab0-53bee3e43ed2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.243071 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.242981    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef79a10-292b-42d9-aab0-53bee3e43ed2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.243126 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.243087    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef79a10-292b-42d9-aab0-53bee3e43ed2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.243185 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.243127    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef79a10-292b-42d9-aab0-53bee3e43ed2-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.243185 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.243146    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef79a10-292b-42d9-aab0-53bee3e43ed2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.243185 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.243164    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8ef79a10-292b-42d9-aab0-53bee3e43ed2-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.243328 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.243225    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8ef79a10-292b-42d9-aab0-53bee3e43ed2-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.243328 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.243282    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef79a10-292b-42d9-aab0-53bee3e43ed2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.243425 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.243380    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dggp4\" (UniqueName: \"kubernetes.io/projected/8ef79a10-292b-42d9-aab0-53bee3e43ed2-kube-api-access-dggp4\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.243631 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.243605    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8ef79a10-292b-42d9-aab0-53bee3e43ed2-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.243771 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.243755    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8ef79a10-292b-42d9-aab0-53bee3e43ed2-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.245356 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.245328    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8ef79a10-292b-42d9-aab0-53bee3e43ed2-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.245678 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.245656    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8ef79a10-292b-42d9-aab0-53bee3e43ed2-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.267209 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.267182    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8ef79a10-292b-42d9-aab0-53bee3e43ed2-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.278224 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.278200    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dggp4\" (UniqueName: \"kubernetes.io/projected/8ef79a10-292b-42d9-aab0-53bee3e43ed2-kube-api-access-dggp4\") pod \"maas-default-gateway-openshift-default-76f4876667-5knvx\" (UID: \"8ef79a10-292b-42d9-aab0-53bee3e43ed2\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.338484 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.338395    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:20.554346 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.554303    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:20.680575 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.680547    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"]
Nov 02 17:46:20.681600 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:46:20.681553    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8ef79a10_292b_42d9_aab0_53bee3e43ed2.slice/crio-aa41378655845156f20a82150f9c692522fe267b901d05ceffa6bd274351ce59 WatchSource:0}: Error finding container aa41378655845156f20a82150f9c692522fe267b901d05ceffa6bd274351ce59: Status 404 returned error can't find the container with id aa41378655845156f20a82150f9c692522fe267b901d05ceffa6bd274351ce59
Nov 02 17:46:20.684636 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.684595    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:20.684735 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.684684    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:20.684735 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.684720    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:46:20.929282 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:20.929135    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:21.092368 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:21.083078    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:21.525091 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:21.525046    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx" event={"ID":"8ef79a10-292b-42d9-aab0-53bee3e43ed2","Type":"ContainerStarted","Data":"b909de065441c6bdf8ea1ba34bd67e2ab5f23fbf455e85700fb47e14a032b579"}
Nov 02 17:46:21.525266 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:21.525149    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx" event={"ID":"8ef79a10-292b-42d9-aab0-53bee3e43ed2","Type":"ContainerStarted","Data":"aa41378655845156f20a82150f9c692522fe267b901d05ceffa6bd274351ce59"}
Nov 02 17:46:21.588469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:21.588394    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx" podStartSLOduration=2.588368898 podStartE2EDuration="2.588368898s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:21.586011159 +0000 UTC m=+998.077074348" watchObservedRunningTime="2026-11-02 17:46:21.588368898 +0000 UTC m=+998.079432061"
Nov 02 17:46:21.623153 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:21.623112    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="954cad70-ab73-4391-b7f4-4fc0d0d23405" path="/var/lib/kubelet/pods/954cad70-ab73-4391-b7f4-4fc0d0d23405/volumes"
Nov 02 17:46:22.339504 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:22.339467    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:22.340772 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:22.340749    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5knvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:46:22.340899 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:22.340805    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx" podUID="8ef79a10-292b-42d9-aab0-53bee3e43ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 02 17:46:22.771520 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:22.771478    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:22.950868 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:22.949968    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:23.339628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:23.339589    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5knvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:46:23.340066 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:23.339644    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx" podUID="8ef79a10-292b-42d9-aab0-53bee3e43ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 02 17:46:24.050426 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:24.050386    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"]
Nov 02 17:46:24.061857 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:24.061829    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4e564800-rzxqj"]
Nov 02 17:46:24.338807 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:24.338716    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5knvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:46:24.338807 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:24.338767    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx" podUID="8ef79a10-292b-42d9-aab0-53bee3e43ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 02 17:46:24.612893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:24.611034    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:24.773622 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:24.773587    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:24.941326 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:24.941291    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:25.343843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:25.343732    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5knvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:46:25.345365 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:25.345329    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx" podUID="8ef79a10-292b-42d9-aab0-53bee3e43ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 02 17:46:25.649129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:25.649044    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="739ac036-8e8a-4585-b03e-febe64f2f54b" path="/var/lib/kubelet/pods/739ac036-8e8a-4585-b03e-febe64f2f54b/volumes"
Nov 02 17:46:25.713175 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:25.709325    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:26.339765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:26.339735    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5knvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:46:26.339939 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:26.339806    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx" podUID="8ef79a10-292b-42d9-aab0-53bee3e43ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 02 17:46:26.522727 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:26.522675    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:26.675787 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:26.675700    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:27.339798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:27.339764    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5knvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:46:27.340018 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:27.339851    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx" podUID="8ef79a10-292b-42d9-aab0-53bee3e43ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 02 17:46:27.665222 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:27.665020    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:28.043802 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:28.043755    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"]
Nov 02 17:46:28.055652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:28.055610    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-9453ae6a-sgnwb"]
Nov 02 17:46:28.070935 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:28.070897    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:28.210653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:28.210607    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"]
Nov 02 17:46:28.215961 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:28.215928    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:28.216691 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:28.216672    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-ait-30a972ba-5h6mk"]
Nov 02 17:46:28.339507 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:28.339420    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5knvx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused" start-of-body=
Nov 02 17:46:28.339507 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:28.339492    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx" podUID="8ef79a10-292b-42d9-aab0-53bee3e43ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": dial tcp 10.134.0.115:15021: connect: connection refused"
Nov 02 17:46:28.849011 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:28.848980    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"]
Nov 02 17:46:28.849229 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:28.849204    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk" podUID="fa32838f-7553-4cf6-adbc-aa7bdada9c0f" containerName="istio-proxy" containerID="cri-o://9459610172c097c21f19e4e84aaf8e60c529796cbce40d873a856cb38b13e69c" gracePeriod=30
Nov 02 17:46:28.945033 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:28.944975    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:29.651627 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:29.651591    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4705529b-3538-410f-9504-b2d8fe857615" path="/var/lib/kubelet/pods/4705529b-3538-410f-9504-b2d8fe857615/volumes"
Nov 02 17:46:29.652729 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:29.652705    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c465e47-a754-42da-b48d-c41acd9fbedf" path="/var/lib/kubelet/pods/8c465e47-a754-42da-b48d-c41acd9fbedf/volumes"
Nov 02 17:46:29.781837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:29.781784    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:30.231926 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:30.231891    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:30.267382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:30.267350    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:30.267734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:30.267710    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:30.269191 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:30.269169    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx"
Nov 02 17:46:31.298769 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:31.298728    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:31.548183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:31.548137    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:31.904843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:31.903172    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:33.890962 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:33.890929    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:34.291854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.276718    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:34.723920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.723345    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:46:34.933900 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.931118    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istiod-ca-cert\") pod \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") "
Nov 02 17:46:34.933900 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.931268    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-token\" (UniqueName: \"kubernetes.io/projected/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-token\") pod \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") "
Nov 02 17:46:34.933900 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.931308    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-workload-socket\") pod \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") "
Nov 02 17:46:34.933900 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.931376    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-envoy\") pod \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") "
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.945291    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istiod-ca-cert" pod "fa32838f-7553-4cf6-adbc-aa7bdada9c0f" (UID: "fa32838f-7553-4cf6-adbc-aa7bdada9c0f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.945411    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-workload-socket" pod "fa32838f-7553-4cf6-adbc-aa7bdada9c0f" (UID: "fa32838f-7553-4cf6-adbc-aa7bdada9c0f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.945531    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-credential-socket\") pod \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") "
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.948317    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-credential-socket" pod "fa32838f-7553-4cf6-adbc-aa7bdada9c0f" (UID: "fa32838f-7553-4cf6-adbc-aa7bdada9c0f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.948448    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-podinfo\") pod \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") "
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.948547    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-kube-api-access-5dx67\" (UniqueName: \"kubernetes.io/projected/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-kube-api-access-5dx67\") pod \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") "
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.948595    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-data\") pod \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") "
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.948629    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-workload-certs\") pod \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\" (UID: \"fa32838f-7553-4cf6-adbc-aa7bdada9c0f\") "
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.950326    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-data" pod "fa32838f-7553-4cf6-adbc-aa7bdada9c0f" (UID: "fa32838f-7553-4cf6-adbc-aa7bdada9c0f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.950572    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-workload-certs" pod "fa32838f-7553-4cf6-adbc-aa7bdada9c0f" (UID: "fa32838f-7553-4cf6-adbc-aa7bdada9c0f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.951139    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.951163    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.951179    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.951193    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.951222    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.954053    4456 generic.go:356] "Generic (PLEG): container finished" podID="fa32838f-7553-4cf6-adbc-aa7bdada9c0f" containerID="9459610172c097c21f19e4e84aaf8e60c529796cbce40d873a856cb38b13e69c" exitCode=0
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.954264    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk" event={"ID":"fa32838f-7553-4cf6-adbc-aa7bdada9c0f","Type":"ContainerDied","Data":"9459610172c097c21f19e4e84aaf8e60c529796cbce40d873a856cb38b13e69c"}
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.954340    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk" event={"ID":"fa32838f-7553-4cf6-adbc-aa7bdada9c0f","Type":"ContainerDied","Data":"9f4522c874b7fa537a3f69697667033adbe6f59d88b9aadc1bab11bd7e6af39d"}
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.954391    4456 scope.go:122] "RemoveContainer" containerID="9459610172c097c21f19e4e84aaf8e60c529796cbce40d873a856cb38b13e69c"
Nov 02 17:46:34.968448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.954603    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"
Nov 02 17:46:34.999296 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.971982    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-envoy" pod "fa32838f-7553-4cf6-adbc-aa7bdada9c0f" (UID: "fa32838f-7553-4cf6-adbc-aa7bdada9c0f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:46:34.999296 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.974012    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-podinfo" pod "fa32838f-7553-4cf6-adbc-aa7bdada9c0f" (UID: "fa32838f-7553-4cf6-adbc-aa7bdada9c0f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:46:34.999296 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.974243    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-token" pod "fa32838f-7553-4cf6-adbc-aa7bdada9c0f" (UID: "fa32838f-7553-4cf6-adbc-aa7bdada9c0f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:46:34.999296 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:34.984653    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-kube-api-access-5dx67" pod "fa32838f-7553-4cf6-adbc-aa7bdada9c0f" (UID: "fa32838f-7553-4cf6-adbc-aa7bdada9c0f"). InnerVolumeSpecName "kube-api-access-5dx67". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:46:35.125842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:35.055754    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5dx67\" (UniqueName: \"kubernetes.io/projected/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-kube-api-access-5dx67\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:35.125842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:35.055807    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:35.125842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:35.055864    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:35.125842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:35.055878    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fa32838f-7553-4cf6-adbc-aa7bdada9c0f-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:46:35.172855 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:35.169983    4456 scope.go:122] "RemoveContainer" containerID="9459610172c097c21f19e4e84aaf8e60c529796cbce40d873a856cb38b13e69c"
Nov 02 17:46:35.200891 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:46:35.199682    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9459610172c097c21f19e4e84aaf8e60c529796cbce40d873a856cb38b13e69c\": container with ID starting with 9459610172c097c21f19e4e84aaf8e60c529796cbce40d873a856cb38b13e69c not found: ID does not exist" containerID="9459610172c097c21f19e4e84aaf8e60c529796cbce40d873a856cb38b13e69c"
Nov 02 17:46:35.200891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:35.199736    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9459610172c097c21f19e4e84aaf8e60c529796cbce40d873a856cb38b13e69c"} err="failed to get container status \"9459610172c097c21f19e4e84aaf8e60c529796cbce40d873a856cb38b13e69c\": rpc error: code = NotFound desc = could not find container \"9459610172c097c21f19e4e84aaf8e60c529796cbce40d873a856cb38b13e69c\": container with ID starting with 9459610172c097c21f19e4e84aaf8e60c529796cbce40d873a856cb38b13e69c not found: ID does not exist"
Nov 02 17:46:35.477708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:35.477623    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"]
Nov 02 17:46:35.528850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:35.524569    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-ait-30a972ba-openshift-default-7d9ccf84bc-9snnk"]
Nov 02 17:46:35.747896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:35.731539    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fa32838f-7553-4cf6-adbc-aa7bdada9c0f" path="/var/lib/kubelet/pods/fa32838f-7553-4cf6-adbc-aa7bdada9c0f/volumes"
Nov 02 17:46:36.286971 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:36.286933    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:36.635912 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:36.635784    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:38.269765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.269728    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:38.432414 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.432363    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:38.618936 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.608413    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:38.682112 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.669925    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"]
Nov 02 17:46:38.736514 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.708333    4456 cpu_manager.go:414] "containerMap: removing container" podUID="fa32838f-7553-4cf6-adbc-aa7bdada9c0f" containerName="istio-proxy"
Nov 02 17:46:38.736514 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.708366    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fa32838f-7553-4cf6-adbc-aa7bdada9c0f" containerName="istio-proxy"
Nov 02 17:46:38.736514 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.708642    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="fa32838f-7553-4cf6-adbc-aa7bdada9c0f" containerName="istio-proxy"
Nov 02 17:46:38.736514 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.732932    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"]
Nov 02 17:46:38.736514 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.733954    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 17:46:38.768842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.768130    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-b91583-dockercfg-vbwks\""
Nov 02 17:46:38.768842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.768282    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-b91583\""
Nov 02 17:46:38.893096 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.892994    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-vj66v\" (UniqueName: \"kubernetes.io/projected/d837dde2-3939-4400-8ea7-fee58c162983-kube-api-access-vj66v\") pod \"payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx\" (UID: \"d837dde2-3939-4400-8ea7-fee58c162983\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 17:46:38.893096 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.893057    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d837dde2-3939-4400-8ea7-fee58c162983-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx\" (UID: \"d837dde2-3939-4400-8ea7-fee58c162983\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 17:46:38.995433 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.995394    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-vj66v\" (UniqueName: \"kubernetes.io/projected/d837dde2-3939-4400-8ea7-fee58c162983-kube-api-access-vj66v\") pod \"payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx\" (UID: \"d837dde2-3939-4400-8ea7-fee58c162983\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 17:46:38.995645 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.995456    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d837dde2-3939-4400-8ea7-fee58c162983-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx\" (UID: \"d837dde2-3939-4400-8ea7-fee58c162983\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 17:46:38.996478 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:38.996449    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d837dde2-3939-4400-8ea7-fee58c162983-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx\" (UID: \"d837dde2-3939-4400-8ea7-fee58c162983\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 17:46:39.021155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:39.021128    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-vj66v\" (UniqueName: \"kubernetes.io/projected/d837dde2-3939-4400-8ea7-fee58c162983-kube-api-access-vj66v\") pod \"payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx\" (UID: \"d837dde2-3939-4400-8ea7-fee58c162983\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 17:46:39.098562 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:39.098513    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 17:46:39.460232 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:39.460058    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"]
Nov 02 17:46:40.023553 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:40.023509    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" event={"ID":"d837dde2-3939-4400-8ea7-fee58c162983","Type":"ContainerStarted","Data":"17cc61a46dc30e87845b6d1ac12d9771996840ced93cb394cfdbcb29883e8285"}
Nov 02 17:46:40.023753 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:40.023582    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" event={"ID":"d837dde2-3939-4400-8ea7-fee58c162983","Type":"ContainerStarted","Data":"06c8e45e6152b3243b57e5eff85877ff6128978eed88c9a57087d53841ecd736"}
Nov 02 17:46:40.024668 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:40.024435    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 17:46:40.074913 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:40.074863    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" podStartSLOduration=2.074845951 podStartE2EDuration="2.074845951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:46:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:46:40.066107826 +0000 UTC m=+1016.557171001" watchObservedRunningTime="2026-11-02 17:46:40.074845951 +0000 UTC m=+1016.565909103"
Nov 02 17:46:40.248915 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:40.248872    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:42.024068 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:42.024026    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:42.176513 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:42.176375    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:43.980836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:43.977225    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:44.018925 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:44.018893    4456 scope.go:122] "RemoveContainer" containerID="8c6f0412d220391d3a2f9412595ab8c617a940b75a1a6eddce8ad00121b6934c"
Nov 02 17:46:44.031338 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:44.031311    4456 scope.go:122] "RemoveContainer" containerID="b0a3d6acd937a153cbcf8537f2fe0c5fffc4dfdbd6773912c0f04f13722892f1"
Nov 02 17:46:45.759389 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:45.759354    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:46.166702 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:46:46.166616    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:46:47.654106 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:47.654050    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:49.402118 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:49.402077    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:51.038860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:51.038798    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 17:46:51.137094 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:51.137055    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:52.939313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:52.939275    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:54.716724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:54.716691    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:54.876220 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:54.876180    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:55.028697 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:55.028574    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:56.194848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:56.194774    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:56.395163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:56.395131    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:56.772535 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:56.772498    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:56.929581 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:56.929541    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:46:58.676124 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:46:58.676088    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:00.464558 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:00.464513    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:02.476706 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:02.476624    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:04.254849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:04.253696    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:05.624511 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:05.624481    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"]
Nov 02 17:47:05.630527 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:05.630505    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"
Nov 02 17:47:05.644539 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:05.644516    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"]
Nov 02 17:47:05.649376 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:05.649352    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:47:05.649623 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:05.649599    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:47:05.719702 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:05.719669    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ea104436-g452t\" (UID: \"f4a5e7a9-67c3-469a-9409-a0b05cb9b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"
Nov 02 17:47:05.719958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:05.719931    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ea104436-g452t\" (UID: \"f4a5e7a9-67c3-469a-9409-a0b05cb9b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"
Nov 02 17:47:05.720345 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:05.720317    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wzbds\" (UniqueName: \"kubernetes.io/projected/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-kube-api-access-wzbds\") pod \"maas-api-revoke-keys-e2e-migrate-ea104436-g452t\" (UID: \"f4a5e7a9-67c3-469a-9409-a0b05cb9b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"
Nov 02 17:47:05.821988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:05.821952    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wzbds\" (UniqueName: \"kubernetes.io/projected/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-kube-api-access-wzbds\") pod \"maas-api-revoke-keys-e2e-migrate-ea104436-g452t\" (UID: \"f4a5e7a9-67c3-469a-9409-a0b05cb9b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"
Nov 02 17:47:05.822164 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:05.822048    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ea104436-g452t\" (UID: \"f4a5e7a9-67c3-469a-9409-a0b05cb9b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"
Nov 02 17:47:05.822293 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:05.822271    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ea104436-g452t\" (UID: \"f4a5e7a9-67c3-469a-9409-a0b05cb9b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"
Nov 02 17:47:05.822752 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:05.822716    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ea104436-g452t\" (UID: \"f4a5e7a9-67c3-469a-9409-a0b05cb9b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"
Nov 02 17:47:05.824946 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:05.824920    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-migrate-ea104436-g452t\" (UID: \"f4a5e7a9-67c3-469a-9409-a0b05cb9b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"
Nov 02 17:47:05.846404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:05.846381    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wzbds\" (UniqueName: \"kubernetes.io/projected/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-kube-api-access-wzbds\") pod \"maas-api-revoke-keys-e2e-migrate-ea104436-g452t\" (UID: \"f4a5e7a9-67c3-469a-9409-a0b05cb9b794\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"
Nov 02 17:47:05.947873 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:05.947844    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"
Nov 02 17:47:06.208656 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:06.208561    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:06.272827 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:06.271010    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"]
Nov 02 17:47:06.272827 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:47:06.272118    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf4a5e7a9_67c3_469a_9409_a0b05cb9b794.slice/crio-7dfd41212db16dc1c557feda0494efa3b0ac44cc054956eb735c675edeb383ed WatchSource:0}: Error finding container 7dfd41212db16dc1c557feda0494efa3b0ac44cc054956eb735c675edeb383ed: Status 404 returned error can't find the container with id 7dfd41212db16dc1c557feda0494efa3b0ac44cc054956eb735c675edeb383ed
Nov 02 17:47:06.357586 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:06.357543    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t" event={"ID":"f4a5e7a9-67c3-469a-9409-a0b05cb9b794","Type":"ContainerStarted","Data":"7dfd41212db16dc1c557feda0494efa3b0ac44cc054956eb735c675edeb383ed"}
Nov 02 17:47:07.077561 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:07.077524    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:07.377050 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:07.376949    4456 generic.go:356] "Generic (PLEG): container finished" podID="f4a5e7a9-67c3-469a-9409-a0b05cb9b794" containerID="ca36d520ce7140765fc905f1273ef62ab673d6854b596a5f11b5a47473a12e51" exitCode=0
Nov 02 17:47:07.377222 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:07.377125    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t" event={"ID":"f4a5e7a9-67c3-469a-9409-a0b05cb9b794","Type":"ContainerDied","Data":"ca36d520ce7140765fc905f1273ef62ab673d6854b596a5f11b5a47473a12e51"}
Nov 02 17:47:08.110855 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.110786    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:08.401421 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.401388    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"]
Nov 02 17:47:08.407602 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.407573    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"
Nov 02 17:47:08.426931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.426899    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"]
Nov 02 17:47:08.476979 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.476945    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77550e6-ffaf-4d67-9748-9ae9ce295727-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-93321a4f-p67rg\" (UID: \"d77550e6-ffaf-4d67-9748-9ae9ce295727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"
Nov 02 17:47:08.477129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.477093    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6s2l\" (UniqueName: \"kubernetes.io/projected/d77550e6-ffaf-4d67-9748-9ae9ce295727-kube-api-access-p6s2l\") pod \"maas-api-revoke-keys-e2e-derive-93321a4f-p67rg\" (UID: \"d77550e6-ffaf-4d67-9748-9ae9ce295727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"
Nov 02 17:47:08.477213 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.477197    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77550e6-ffaf-4d67-9748-9ae9ce295727-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-93321a4f-p67rg\" (UID: \"d77550e6-ffaf-4d67-9748-9ae9ce295727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"
Nov 02 17:47:08.578827 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.578780    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77550e6-ffaf-4d67-9748-9ae9ce295727-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-93321a4f-p67rg\" (UID: \"d77550e6-ffaf-4d67-9748-9ae9ce295727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"
Nov 02 17:47:08.578991 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.578977    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77550e6-ffaf-4d67-9748-9ae9ce295727-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-93321a4f-p67rg\" (UID: \"d77550e6-ffaf-4d67-9748-9ae9ce295727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"
Nov 02 17:47:08.579074 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.579052    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6s2l\" (UniqueName: \"kubernetes.io/projected/d77550e6-ffaf-4d67-9748-9ae9ce295727-kube-api-access-p6s2l\") pod \"maas-api-revoke-keys-e2e-derive-93321a4f-p67rg\" (UID: \"d77550e6-ffaf-4d67-9748-9ae9ce295727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"
Nov 02 17:47:08.579758 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.579731    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77550e6-ffaf-4d67-9748-9ae9ce295727-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-derive-93321a4f-p67rg\" (UID: \"d77550e6-ffaf-4d67-9748-9ae9ce295727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"
Nov 02 17:47:08.581430 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.581409    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77550e6-ffaf-4d67-9748-9ae9ce295727-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-derive-93321a4f-p67rg\" (UID: \"d77550e6-ffaf-4d67-9748-9ae9ce295727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"
Nov 02 17:47:08.604052 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.604015    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6s2l\" (UniqueName: \"kubernetes.io/projected/d77550e6-ffaf-4d67-9748-9ae9ce295727-kube-api-access-p6s2l\") pod \"maas-api-revoke-keys-e2e-derive-93321a4f-p67rg\" (UID: \"d77550e6-ffaf-4d67-9748-9ae9ce295727\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"
Nov 02 17:47:08.627432 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.627399    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"
Nov 02 17:47:08.680280 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.680190    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-maas-api-openshift-ca\") pod \"f4a5e7a9-67c3-469a-9409-a0b05cb9b794\" (UID: \"f4a5e7a9-67c3-469a-9409-a0b05cb9b794\") "
Nov 02 17:47:08.680454 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.680302    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-kube-api-access-wzbds\" (UniqueName: \"kubernetes.io/projected/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-kube-api-access-wzbds\") pod \"f4a5e7a9-67c3-469a-9409-a0b05cb9b794\" (UID: \"f4a5e7a9-67c3-469a-9409-a0b05cb9b794\") "
Nov 02 17:47:08.680454 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.680374    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-maas-api-xks-ca\") pod \"f4a5e7a9-67c3-469a-9409-a0b05cb9b794\" (UID: \"f4a5e7a9-67c3-469a-9409-a0b05cb9b794\") "
Nov 02 17:47:08.680623 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.680587    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-maas-api-openshift-ca" pod "f4a5e7a9-67c3-469a-9409-a0b05cb9b794" (UID: "f4a5e7a9-67c3-469a-9409-a0b05cb9b794"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:47:08.681229 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.681203    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:08.683865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.683799    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-kube-api-access-wzbds" pod "f4a5e7a9-67c3-469a-9409-a0b05cb9b794" (UID: "f4a5e7a9-67c3-469a-9409-a0b05cb9b794"). InnerVolumeSpecName "kube-api-access-wzbds". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:47:08.685242 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.685215    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-maas-api-xks-ca" pod "f4a5e7a9-67c3-469a-9409-a0b05cb9b794" (UID: "f4a5e7a9-67c3-469a-9409-a0b05cb9b794"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:47:08.731927 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.731888    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"
Nov 02 17:47:08.783207 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.783050    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzbds\" (UniqueName: \"kubernetes.io/projected/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-kube-api-access-wzbds\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:08.783207 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:08.783087    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f4a5e7a9-67c3-469a-9409-a0b05cb9b794-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:09.079219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:09.079182    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"]
Nov 02 17:47:09.399631 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:09.399543    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"
Nov 02 17:47:09.400110 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:09.399546    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t" event={"ID":"f4a5e7a9-67c3-469a-9409-a0b05cb9b794","Type":"ContainerDied","Data":"7dfd41212db16dc1c557feda0494efa3b0ac44cc054956eb735c675edeb383ed"}
Nov 02 17:47:09.400110 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:09.399678    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="7dfd41212db16dc1c557feda0494efa3b0ac44cc054956eb735c675edeb383ed"
Nov 02 17:47:09.401411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:09.401387    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg" event={"ID":"d77550e6-ffaf-4d67-9748-9ae9ce295727","Type":"ContainerStarted","Data":"ba228041c606d617f36975a299b3214347b787bce548c3a6f09c7e136ad50f2a"}
Nov 02 17:47:09.909282 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:09.909239    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:10.415952 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:10.415913    4456 generic.go:356] "Generic (PLEG): container finished" podID="d77550e6-ffaf-4d67-9748-9ae9ce295727" containerID="8840ba33147f6d7c04b50c022624be7e756f0b168b4f6c6c496dc8e5f954fe01" exitCode=0
Nov 02 17:47:10.416423 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:10.416074    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg" event={"ID":"d77550e6-ffaf-4d67-9748-9ae9ce295727","Type":"ContainerDied","Data":"8840ba33147f6d7c04b50c022624be7e756f0b168b4f6c6c496dc8e5f954fe01"}
Nov 02 17:47:11.544241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.544072    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"]
Nov 02 17:47:11.545136 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.544827    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f4a5e7a9-67c3-469a-9409-a0b05cb9b794" containerName="revoke-keys"
Nov 02 17:47:11.545136 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.544845    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4a5e7a9-67c3-469a-9409-a0b05cb9b794" containerName="revoke-keys"
Nov 02 17:47:11.545136 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.544953    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4a5e7a9-67c3-469a-9409-a0b05cb9b794" containerName="revoke-keys"
Nov 02 17:47:11.556762 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.548950    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"
Nov 02 17:47:11.577172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.577145    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"]
Nov 02 17:47:11.626880 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.625256    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51884bda-c8ca-483a-b9da-f85e5624c1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94e3ead7-bhrxm\" (UID: \"51884bda-c8ca-483a-b9da-f85e5624c1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"
Nov 02 17:47:11.626880 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.625461    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51884bda-c8ca-483a-b9da-f85e5624c1c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94e3ead7-bhrxm\" (UID: \"51884bda-c8ca-483a-b9da-f85e5624c1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"
Nov 02 17:47:11.626880 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.625753    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8ncgq\" (UniqueName: \"kubernetes.io/projected/51884bda-c8ca-483a-b9da-f85e5624c1c2-kube-api-access-8ncgq\") pod \"maas-api-revoke-sub-94e3ead7-bhrxm\" (UID: \"51884bda-c8ca-483a-b9da-f85e5624c1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"
Nov 02 17:47:11.645379 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.645354    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"
Nov 02 17:47:11.727631 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.727575    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d77550e6-ffaf-4d67-9748-9ae9ce295727-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77550e6-ffaf-4d67-9748-9ae9ce295727-maas-api-openshift-ca\") pod \"d77550e6-ffaf-4d67-9748-9ae9ce295727\" (UID: \"d77550e6-ffaf-4d67-9748-9ae9ce295727\") "
Nov 02 17:47:11.727864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.727685    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d77550e6-ffaf-4d67-9748-9ae9ce295727-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77550e6-ffaf-4d67-9748-9ae9ce295727-maas-api-xks-ca\") pod \"d77550e6-ffaf-4d67-9748-9ae9ce295727\" (UID: \"d77550e6-ffaf-4d67-9748-9ae9ce295727\") "
Nov 02 17:47:11.727864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.727775    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d77550e6-ffaf-4d67-9748-9ae9ce295727-kube-api-access-p6s2l\" (UniqueName: \"kubernetes.io/projected/d77550e6-ffaf-4d67-9748-9ae9ce295727-kube-api-access-p6s2l\") pod \"d77550e6-ffaf-4d67-9748-9ae9ce295727\" (UID: \"d77550e6-ffaf-4d67-9748-9ae9ce295727\") "
Nov 02 17:47:11.728083 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.728051    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d77550e6-ffaf-4d67-9748-9ae9ce295727-maas-api-openshift-ca" pod "d77550e6-ffaf-4d67-9748-9ae9ce295727" (UID: "d77550e6-ffaf-4d67-9748-9ae9ce295727"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:47:11.728250 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.728223    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51884bda-c8ca-483a-b9da-f85e5624c1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94e3ead7-bhrxm\" (UID: \"51884bda-c8ca-483a-b9da-f85e5624c1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"
Nov 02 17:47:11.728396 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.728379    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51884bda-c8ca-483a-b9da-f85e5624c1c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94e3ead7-bhrxm\" (UID: \"51884bda-c8ca-483a-b9da-f85e5624c1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"
Nov 02 17:47:11.728600 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.728580    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8ncgq\" (UniqueName: \"kubernetes.io/projected/51884bda-c8ca-483a-b9da-f85e5624c1c2-kube-api-access-8ncgq\") pod \"maas-api-revoke-sub-94e3ead7-bhrxm\" (UID: \"51884bda-c8ca-483a-b9da-f85e5624c1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"
Nov 02 17:47:11.728735 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.728679    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d77550e6-ffaf-4d67-9748-9ae9ce295727-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:11.729389 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.729359    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51884bda-c8ca-483a-b9da-f85e5624c1c2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-94e3ead7-bhrxm\" (UID: \"51884bda-c8ca-483a-b9da-f85e5624c1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"
Nov 02 17:47:11.730941 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.730914    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d77550e6-ffaf-4d67-9748-9ae9ce295727-kube-api-access-p6s2l" pod "d77550e6-ffaf-4d67-9748-9ae9ce295727" (UID: "d77550e6-ffaf-4d67-9748-9ae9ce295727"). InnerVolumeSpecName "kube-api-access-p6s2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:47:11.731745 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.731719    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d77550e6-ffaf-4d67-9748-9ae9ce295727-maas-api-xks-ca" pod "d77550e6-ffaf-4d67-9748-9ae9ce295727" (UID: "d77550e6-ffaf-4d67-9748-9ae9ce295727"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:47:11.732466 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.732442    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51884bda-c8ca-483a-b9da-f85e5624c1c2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-94e3ead7-bhrxm\" (UID: \"51884bda-c8ca-483a-b9da-f85e5624c1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"
Nov 02 17:47:11.753407 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.753317    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8ncgq\" (UniqueName: \"kubernetes.io/projected/51884bda-c8ca-483a-b9da-f85e5624c1c2-kube-api-access-8ncgq\") pod \"maas-api-revoke-sub-94e3ead7-bhrxm\" (UID: \"51884bda-c8ca-483a-b9da-f85e5624c1c2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"
Nov 02 17:47:11.829873 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.829806    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d77550e6-ffaf-4d67-9748-9ae9ce295727-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:11.829873 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.829874    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6s2l\" (UniqueName: \"kubernetes.io/projected/d77550e6-ffaf-4d67-9748-9ae9ce295727-kube-api-access-p6s2l\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:11.874382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.874349    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"
Nov 02 17:47:11.907043 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:11.907001    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:12.213405 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:12.213359    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"]
Nov 02 17:47:12.213701 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:47:12.213675    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod51884bda_c8ca_483a_b9da_f85e5624c1c2.slice/crio-aa6c9faf49eb7d3d10e20b6996db9492e1aa3d563c26e97f20ae627c2a9a3e81 WatchSource:0}: Error finding container aa6c9faf49eb7d3d10e20b6996db9492e1aa3d563c26e97f20ae627c2a9a3e81: Status 404 returned error can't find the container with id aa6c9faf49eb7d3d10e20b6996db9492e1aa3d563c26e97f20ae627c2a9a3e81
Nov 02 17:47:12.439207 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:12.439179    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"
Nov 02 17:47:12.439383 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:12.439220    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg" event={"ID":"d77550e6-ffaf-4d67-9748-9ae9ce295727","Type":"ContainerDied","Data":"ba228041c606d617f36975a299b3214347b787bce548c3a6f09c7e136ad50f2a"}
Nov 02 17:47:12.439383 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:12.439271    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba228041c606d617f36975a299b3214347b787bce548c3a6f09c7e136ad50f2a"
Nov 02 17:47:12.440769 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:12.440746    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm" event={"ID":"51884bda-c8ca-483a-b9da-f85e5624c1c2","Type":"ContainerStarted","Data":"aa6c9faf49eb7d3d10e20b6996db9492e1aa3d563c26e97f20ae627c2a9a3e81"}
Nov 02 17:47:13.450122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:13.450090    4456 generic.go:356] "Generic (PLEG): container finished" podID="51884bda-c8ca-483a-b9da-f85e5624c1c2" containerID="343e10261cea507061afeab13d56732972ca7a49e6face19ff3c2566f889aa49" exitCode=0
Nov 02 17:47:13.450520 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:13.450181    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm" event={"ID":"51884bda-c8ca-483a-b9da-f85e5624c1c2","Type":"ContainerDied","Data":"343e10261cea507061afeab13d56732972ca7a49e6face19ff3c2566f889aa49"}
Nov 02 17:47:13.686630 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:13.686599    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:14.646924 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:14.646893    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"
Nov 02 17:47:14.679302 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:14.679277    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/51884bda-c8ca-483a-b9da-f85e5624c1c2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51884bda-c8ca-483a-b9da-f85e5624c1c2-maas-api-xks-ca\") pod \"51884bda-c8ca-483a-b9da-f85e5624c1c2\" (UID: \"51884bda-c8ca-483a-b9da-f85e5624c1c2\") "
Nov 02 17:47:14.679456 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:14.679316    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/51884bda-c8ca-483a-b9da-f85e5624c1c2-kube-api-access-8ncgq\" (UniqueName: \"kubernetes.io/projected/51884bda-c8ca-483a-b9da-f85e5624c1c2-kube-api-access-8ncgq\") pod \"51884bda-c8ca-483a-b9da-f85e5624c1c2\" (UID: \"51884bda-c8ca-483a-b9da-f85e5624c1c2\") "
Nov 02 17:47:14.679563 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:14.679544    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/51884bda-c8ca-483a-b9da-f85e5624c1c2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51884bda-c8ca-483a-b9da-f85e5624c1c2-maas-api-openshift-ca\") pod \"51884bda-c8ca-483a-b9da-f85e5624c1c2\" (UID: \"51884bda-c8ca-483a-b9da-f85e5624c1c2\") "
Nov 02 17:47:14.679878 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:14.679847    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/51884bda-c8ca-483a-b9da-f85e5624c1c2-maas-api-openshift-ca" pod "51884bda-c8ca-483a-b9da-f85e5624c1c2" (UID: "51884bda-c8ca-483a-b9da-f85e5624c1c2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:47:14.680534 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:14.680418    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/51884bda-c8ca-483a-b9da-f85e5624c1c2-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:14.681787 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:14.681760    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/51884bda-c8ca-483a-b9da-f85e5624c1c2-kube-api-access-8ncgq" pod "51884bda-c8ca-483a-b9da-f85e5624c1c2" (UID: "51884bda-c8ca-483a-b9da-f85e5624c1c2"). InnerVolumeSpecName "kube-api-access-8ncgq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:47:14.681936 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:14.681918    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/51884bda-c8ca-483a-b9da-f85e5624c1c2-maas-api-xks-ca" pod "51884bda-c8ca-483a-b9da-f85e5624c1c2" (UID: "51884bda-c8ca-483a-b9da-f85e5624c1c2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:47:14.782495 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:14.782414    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/51884bda-c8ca-483a-b9da-f85e5624c1c2-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:14.782495 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:14.782448    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8ncgq\" (UniqueName: \"kubernetes.io/projected/51884bda-c8ca-483a-b9da-f85e5624c1c2-kube-api-access-8ncgq\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:15.464038 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:15.464002    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:15.467595 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:15.467569    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"
Nov 02 17:47:15.467757 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:15.467587    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm" event={"ID":"51884bda-c8ca-483a-b9da-f85e5624c1c2","Type":"ContainerDied","Data":"aa6c9faf49eb7d3d10e20b6996db9492e1aa3d563c26e97f20ae627c2a9a3e81"}
Nov 02 17:47:15.467757 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:15.467669    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="aa6c9faf49eb7d3d10e20b6996db9492e1aa3d563c26e97f20ae627c2a9a3e81"
Nov 02 17:47:15.677056 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:15.677026    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:15.858065 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:15.857964    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:16.005953 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:16.005916    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:16.166898 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:47:16.166793    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:47:16.390197 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:16.390157    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:16.543795 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:16.543695    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:17.388436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:17.388406    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:19.095173 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.095135    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:19.280720 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.280683    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:19.320330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.320295    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"]
Nov 02 17:47:19.321355 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.321329    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d77550e6-ffaf-4d67-9748-9ae9ce295727" containerName="revoke-keys"
Nov 02 17:47:19.321355 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.321354    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d77550e6-ffaf-4d67-9748-9ae9ce295727" containerName="revoke-keys"
Nov 02 17:47:19.321578 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.321439    4456 cpu_manager.go:414] "containerMap: removing container" podUID="51884bda-c8ca-483a-b9da-f85e5624c1c2" containerName="revoke-keys"
Nov 02 17:47:19.321578 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.321448    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="51884bda-c8ca-483a-b9da-f85e5624c1c2" containerName="revoke-keys"
Nov 02 17:47:19.321694 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.321622    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d77550e6-ffaf-4d67-9748-9ae9ce295727" containerName="revoke-keys"
Nov 02 17:47:19.321694 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.321646    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="51884bda-c8ca-483a-b9da-f85e5624c1c2" containerName="revoke-keys"
Nov 02 17:47:19.326208 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.326184    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.341708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.341669    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-derive-93321a4f-openshift-default-dockercfg-8fp78\""
Nov 02 17:47:19.363334 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.363255    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"]
Nov 02 17:47:19.479422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.479381    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-workload-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.479603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.479443    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4z54v\" (UniqueName: \"kubernetes.io/projected/f616cbb0-e127-496b-be3c-55181e83ab19-kube-api-access-4z54v\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.479603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.479500    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f616cbb0-e127-496b-be3c-55181e83ab19-istiod-ca-cert\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.479603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.479579    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f616cbb0-e127-496b-be3c-55181e83ab19-istio-podinfo\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.479765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.479662    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f616cbb0-e127-496b-be3c-55181e83ab19-istio-token\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.479892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.479878    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-workload-certs\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.480038 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.480021    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-credential-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.480104 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.480052    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-istio-data\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.480104 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.480083    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-istio-envoy\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.582397 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.582357    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f616cbb0-e127-496b-be3c-55181e83ab19-istio-podinfo\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.582610 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.582468    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f616cbb0-e127-496b-be3c-55181e83ab19-istio-token\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.582610 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.582579    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-workload-certs\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.582749 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.582724    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-credential-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.582749 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.582751    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-istio-data\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.583020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.582791    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-istio-envoy\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.583020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.582942    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-workload-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.583020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.582976    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4z54v\" (UniqueName: \"kubernetes.io/projected/f616cbb0-e127-496b-be3c-55181e83ab19-kube-api-access-4z54v\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.583020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.583017    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f616cbb0-e127-496b-be3c-55181e83ab19-istiod-ca-cert\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.584215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.583645    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-credential-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.584215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.583862    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f616cbb0-e127-496b-be3c-55181e83ab19-istiod-ca-cert\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.584215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.584145    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-workload-socket\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.584669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.584649    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-workload-certs\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.585074 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.585054    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-istio-data\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.617884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.588997    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-istio-envoy\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.617884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.591854    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f616cbb0-e127-496b-be3c-55181e83ab19-istio-podinfo\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.638480 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.638447    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f616cbb0-e127-496b-be3c-55181e83ab19-istio-token\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.643474 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.643444    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4z54v\" (UniqueName: \"kubernetes.io/projected/f616cbb0-e127-496b-be3c-55181e83ab19-kube-api-access-4z54v\") pod \"e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") " pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.661896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.657396    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:19.861838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.860257    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:19.987841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:19.987532    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"]
Nov 02 17:47:20.002113 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:20.002074    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-migrate-ea104436-g452t"]
Nov 02 17:47:20.017573 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:20.013961    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:20.037162 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:20.037133    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"]
Nov 02 17:47:20.037612 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:47:20.037589    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf616cbb0_e127_496b_be3c_55181e83ab19.slice/crio-41717dd259d2550394b183c5308c7cbf5adc1b8acc3cf8dce50eaf3a3d637d4e WatchSource:0}: Error finding container 41717dd259d2550394b183c5308c7cbf5adc1b8acc3cf8dce50eaf3a3d637d4e: Status 404 returned error can't find the container with id 41717dd259d2550394b183c5308c7cbf5adc1b8acc3cf8dce50eaf3a3d637d4e
Nov 02 17:47:20.040444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:20.040414    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:20.040547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:20.040493    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:20.040547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:20.040536    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:20.209043 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:20.208997    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:20.541119 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:20.541079    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc" event={"ID":"f616cbb0-e127-496b-be3c-55181e83ab19","Type":"ContainerStarted","Data":"fa28955233cdf8a45b7a5132372edaddbffc9411d61ad23aeeb3bdd2eae2a2ac"}
Nov 02 17:47:20.541322 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:20.541144    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc" event={"ID":"f616cbb0-e127-496b-be3c-55181e83ab19","Type":"ContainerStarted","Data":"41717dd259d2550394b183c5308c7cbf5adc1b8acc3cf8dce50eaf3a3d637d4e"}
Nov 02 17:47:20.593194 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:20.593136    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc" podStartSLOduration=1.593118548 podStartE2EDuration="1.593118548s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:20.588781538 +0000 UTC m=+1057.079844718" watchObservedRunningTime="2026-11-02 17:47:20.593118548 +0000 UTC m=+1057.084181712"
Nov 02 17:47:20.664223 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:20.664186    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:20.994854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:20.994788    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:21.399142 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.399046    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"]
Nov 02 17:47:21.438463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.438423    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.451363 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.451323    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"]
Nov 02 17:47:21.458639 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.458609    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w5-504de7-openshift-default-dockercfg-f4lbh\""
Nov 02 17:47:21.517831 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.517769    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:21.535165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.535127    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-podinfo\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.535359 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.535276    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-data\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.535359 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.535318    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-workload-certs\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.535481 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.535442    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-token\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.535536 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.535503    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-envoy\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.535601 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.535536    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l2d6n\" (UniqueName: \"kubernetes.io/projected/627671f3-ee4e-4ff9-b03f-235489bc356f-kube-api-access-l2d6n\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.535601 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.535595    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-workload-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.535708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.535622    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-credential-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.535765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.535731    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627671f3-ee4e-4ff9-b03f-235489bc356f-istiod-ca-cert\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.654883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.652417    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-podinfo\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.654883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.652560    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-data\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.654883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.652598    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-workload-certs\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.654883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.652718    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-token\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.654883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.652767    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-envoy\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.654883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.652798    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l2d6n\" (UniqueName: \"kubernetes.io/projected/627671f3-ee4e-4ff9-b03f-235489bc356f-kube-api-access-l2d6n\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.654883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.652881    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-workload-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.654883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.652909    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-credential-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.654883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.653014    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627671f3-ee4e-4ff9-b03f-235489bc356f-istiod-ca-cert\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.702215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.678730    4456 patch_prober.go:29] interesting pod/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 17:47:21.702215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.678785    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc" podUID="f616cbb0-e127-496b-be3c-55181e83ab19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 17:47:21.702215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.679081    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627671f3-ee4e-4ff9-b03f-235489bc356f-istiod-ca-cert\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.702215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.679417    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-data\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.702215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.679688    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-workload-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.702215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.680234    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-workload-certs\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.702215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.680484    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-credential-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.705325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.705286    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:21.785917 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.777805    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4a5e7a9-67c3-469a-9409-a0b05cb9b794" path="/var/lib/kubelet/pods/f4a5e7a9-67c3-469a-9409-a0b05cb9b794/volumes"
Nov 02 17:47:21.813842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.810137    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-token\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.813842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.813034    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-podinfo\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.813842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.813442    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-envoy\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.817167 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.817144    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l2d6n\" (UniqueName: \"kubernetes.io/projected/627671f3-ee4e-4ff9-b03f-235489bc356f-kube-api-access-l2d6n\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:21.921669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:21.921570    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:22.072840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:22.070954    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:22.074032 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:22.073954    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:22.245873 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:22.245725    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:22.436170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:22.436132    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:22.644845 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:47:22.641312    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod627671f3_ee4e_4ff9_b03f_235489bc356f.slice/crio-708d50b71df2a3bb0fe9fa5d1c85353d7c1b0d39fd39683dbf0a12a97a845e9e WatchSource:0}: Error finding container 708d50b71df2a3bb0fe9fa5d1c85353d7c1b0d39fd39683dbf0a12a97a845e9e: Status 404 returned error can't find the container with id 708d50b71df2a3bb0fe9fa5d1c85353d7c1b0d39fd39683dbf0a12a97a845e9e
Nov 02 17:47:22.650841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:22.649854    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"]
Nov 02 17:47:22.655437 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:22.654801    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:22.655437 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:22.654886    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:22.655437 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:22.654917    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:22.668053 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:22.668027    4456 patch_prober.go:29] interesting pod/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 17:47:22.668458 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:22.668432    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc" podUID="f616cbb0-e127-496b-be3c-55181e83ab19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 17:47:22.788347 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:22.754586    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:22.969196 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:22.969042    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:22.976633 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:22.974335    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"]
Nov 02 17:47:23.004620 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:23.004363    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-derive-93321a4f-p67rg"]
Nov 02 17:47:23.166999 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:23.146348    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:23.190841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:23.187760    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"]
Nov 02 17:47:23.467875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:23.457072    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:23.700870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:23.689683    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc" podUID="f616cbb0-e127-496b-be3c-55181e83ab19" containerName="istio-proxy" containerID="cri-o://fa28955233cdf8a45b7a5132372edaddbffc9411d61ad23aeeb3bdd2eae2a2ac" gracePeriod=30
Nov 02 17:47:23.702052 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:23.702024    4456 patch_prober.go:29] interesting pod/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused" start-of-body=
Nov 02 17:47:23.702457 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:23.702303    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc" podUID="f616cbb0-e127-496b-be3c-55181e83ab19" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.120:15021/healthz/ready\": dial tcp 10.134.0.120:15021: connect: connection refused"
Nov 02 17:47:23.721478 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:23.721392    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d77550e6-ffaf-4d67-9748-9ae9ce295727" path="/var/lib/kubelet/pods/d77550e6-ffaf-4d67-9748-9ae9ce295727/volumes"
Nov 02 17:47:23.722232 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:23.722207    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt" event={"ID":"627671f3-ee4e-4ff9-b03f-235489bc356f","Type":"ContainerStarted","Data":"6f61a792111d21e8b0f4badd27b2c7c04693431f7923f8a8a2d0449f05484e9e"}
Nov 02 17:47:23.722411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:23.722394    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt" event={"ID":"627671f3-ee4e-4ff9-b03f-235489bc356f","Type":"ContainerStarted","Data":"708d50b71df2a3bb0fe9fa5d1c85353d7c1b0d39fd39683dbf0a12a97a845e9e"}
Nov 02 17:47:23.757869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:23.756678    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt" podStartSLOduration=2.756643136 podStartE2EDuration="2.756643136s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:23.750590127 +0000 UTC m=+1060.241653289" watchObservedRunningTime="2026-11-02 17:47:23.756643136 +0000 UTC m=+1060.247706300"
Nov 02 17:47:23.886875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:23.886797    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:24.057014 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:24.056889    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:24.073097 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:24.073064    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:24.074942 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:24.074906    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 17:47:24.075166 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:24.074964    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt" podUID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 17:47:24.884844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:24.882566    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:25.076498 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:25.076470    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:25.372383 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:25.372244    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:25.859692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:25.859657    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:25.959182 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:25.959151    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:25.960570 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:25.960548    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:47:26.352178 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:26.352149    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:26.522695 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:26.522547    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:28.308555 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:28.308524    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:28.456630 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:28.456599    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"]
Nov 02 17:47:28.463040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:28.463013    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 17:47:28.487312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:28.487210    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-504de7-dockercfg-d4zk4\""
Nov 02 17:47:28.493061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:28.493038    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-504de7\""
Nov 02 17:47:28.494280 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:28.494171    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"]
Nov 02 17:47:28.510173 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:28.510137    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:28.593058 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:28.592972    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d5jvw\" (UniqueName: \"kubernetes.io/projected/e4c1132f-05b5-4664-84a2-ece57727b12a-kube-api-access-d5jvw\") pod \"payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8\" (UID: \"e4c1132f-05b5-4664-84a2-ece57727b12a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 17:47:28.593262 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:28.593236    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c1132f-05b5-4664-84a2-ece57727b12a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8\" (UID: \"e4c1132f-05b5-4664-84a2-ece57727b12a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 17:47:28.694941 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:28.694902    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c1132f-05b5-4664-84a2-ece57727b12a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8\" (UID: \"e4c1132f-05b5-4664-84a2-ece57727b12a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 17:47:28.695228 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:28.695203    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d5jvw\" (UniqueName: \"kubernetes.io/projected/e4c1132f-05b5-4664-84a2-ece57727b12a-kube-api-access-d5jvw\") pod \"payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8\" (UID: \"e4c1132f-05b5-4664-84a2-ece57727b12a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 17:47:28.695846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:28.695805    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c1132f-05b5-4664-84a2-ece57727b12a-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8\" (UID: \"e4c1132f-05b5-4664-84a2-ece57727b12a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 17:47:28.719470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:28.719431    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d5jvw\" (UniqueName: \"kubernetes.io/projected/e4c1132f-05b5-4664-84a2-ece57727b12a-kube-api-access-d5jvw\") pod \"payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8\" (UID: \"e4c1132f-05b5-4664-84a2-ece57727b12a\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 17:47:28.808433 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:28.805143    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 17:47:29.195933 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:29.195900    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"]
Nov 02 17:47:29.197272 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:47:29.197242    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode4c1132f_05b5_4664_84a2_ece57727b12a.slice/crio-44e06805a9771401184c6e3698e3e96b09d126e5304a78e131abf3ee46746816 WatchSource:0}: Error finding container 44e06805a9771401184c6e3698e3e96b09d126e5304a78e131abf3ee46746816: Status 404 returned error can't find the container with id 44e06805a9771401184c6e3698e3e96b09d126e5304a78e131abf3ee46746816
Nov 02 17:47:29.547626 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:29.547550    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:29.756218 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:29.756183    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:29.956982 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:29.956949    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:30.036331 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:30.036298    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" event={"ID":"e4c1132f-05b5-4664-84a2-ece57727b12a","Type":"ContainerStarted","Data":"a3c609da08ba303e35942c668188b131f4c241ad4c0f9869076f8590a95a0a00"}
Nov 02 17:47:30.036540 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:30.036351    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" event={"ID":"e4c1132f-05b5-4664-84a2-ece57727b12a","Type":"ContainerStarted","Data":"44e06805a9771401184c6e3698e3e96b09d126e5304a78e131abf3ee46746816"}
Nov 02 17:47:30.036874 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:30.036557    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 17:47:30.092445 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:30.092377    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" podStartSLOduration=2.092361044 podStartE2EDuration="2.092361044s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:30.089649107 +0000 UTC m=+1066.580712268" watchObservedRunningTime="2026-11-02 17:47:30.092361044 +0000 UTC m=+1066.583424206"
Nov 02 17:47:30.188431 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:30.188395    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:30.485162 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:30.485128    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:30.768321 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:30.768235    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:31.306151 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.306119    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:31.372061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.372032    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"]
Nov 02 17:47:31.376701 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.376683    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.393677 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.393653    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w6-b91583-openshift-default-dockercfg-68cp5\""
Nov 02 17:47:31.411249 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.411218    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"]
Nov 02 17:47:31.485993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.485956    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a26856-e0a4-410b-978b-4c65e143a80b-istiod-ca-cert\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.486199 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.486144    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-workload-certs\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.486291 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.486221    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a26856-e0a4-410b-978b-4c65e143a80b-istio-podinfo\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.486384 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.486365    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-istio-envoy\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.486452 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.486427    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a26856-e0a4-410b-978b-4c65e143a80b-istio-token\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.486508 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.486461    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lvkmg\" (UniqueName: \"kubernetes.io/projected/20a26856-e0a4-410b-978b-4c65e143a80b-kube-api-access-lvkmg\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.486568 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.486520    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-istio-data\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.486568 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.486554    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-credential-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.486677 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.486629    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-workload-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.492457 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.492425    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:31.601040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.600945    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a26856-e0a4-410b-978b-4c65e143a80b-istio-token\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.601040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.601007    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lvkmg\" (UniqueName: \"kubernetes.io/projected/20a26856-e0a4-410b-978b-4c65e143a80b-kube-api-access-lvkmg\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.601280 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.601067    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-istio-data\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.601280 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.601098    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-credential-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.601280 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.601169    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-workload-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.601460 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.601289    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a26856-e0a4-410b-978b-4c65e143a80b-istiod-ca-cert\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.601460 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.601394    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-workload-certs\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.601570 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.601460    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a26856-e0a4-410b-978b-4c65e143a80b-istio-podinfo\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.601630 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.601598    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-istio-envoy\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.603715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.603681    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-credential-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.604513 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.604481    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-istio-data\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.612835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.605141    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a26856-e0a4-410b-978b-4c65e143a80b-istiod-ca-cert\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.612835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.605385    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-workload-socket\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.612835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.605606    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-workload-certs\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.649837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.646734    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-istio-envoy\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.680249 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.665857    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a26856-e0a4-410b-978b-4c65e143a80b-istio-podinfo\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.722840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.699423    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a26856-e0a4-410b-978b-4c65e143a80b-istio-token\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.722840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.702667    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lvkmg\" (UniqueName: \"kubernetes.io/projected/20a26856-e0a4-410b-978b-4c65e143a80b-kube-api-access-lvkmg\") pod \"e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") " pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:31.746882 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:31.738168    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:32.192607 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.192576    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"]
Nov 02 17:47:32.193466 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:47:32.193438    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod20a26856_e0a4_410b_978b_4c65e143a80b.slice/crio-2260b77e9d5b669a6eb507e90dbf5b42c7b883d214098d7687befbe44f307a0e WatchSource:0}: Error finding container 2260b77e9d5b669a6eb507e90dbf5b42c7b883d214098d7687befbe44f307a0e: Status 404 returned error can't find the container with id 2260b77e9d5b669a6eb507e90dbf5b42c7b883d214098d7687befbe44f307a0e
Nov 02 17:47:32.196714 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.196679    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:32.196871 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.196762    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:32.196871 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.196808    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:47:32.596339 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.596311    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:32.804622 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.804509    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f616cbb0-e127-496b-be3c-55181e83ab19-kube-api-access-4z54v\" (UniqueName: \"kubernetes.io/projected/f616cbb0-e127-496b-be3c-55181e83ab19-kube-api-access-4z54v\") pod \"f616cbb0-e127-496b-be3c-55181e83ab19\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") "
Nov 02 17:47:32.804622 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.804573    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-workload-socket\") pod \"f616cbb0-e127-496b-be3c-55181e83ab19\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") "
Nov 02 17:47:32.804931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.804635    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-credential-socket\") pod \"f616cbb0-e127-496b-be3c-55181e83ab19\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") "
Nov 02 17:47:32.804931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.804669    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-istio-data\") pod \"f616cbb0-e127-496b-be3c-55181e83ab19\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") "
Nov 02 17:47:32.804931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.804700    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f616cbb0-e127-496b-be3c-55181e83ab19-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f616cbb0-e127-496b-be3c-55181e83ab19-istiod-ca-cert\") pod \"f616cbb0-e127-496b-be3c-55181e83ab19\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") "
Nov 02 17:47:32.804931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.804845    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f616cbb0-e127-496b-be3c-55181e83ab19-istio-token\" (UniqueName: \"kubernetes.io/projected/f616cbb0-e127-496b-be3c-55181e83ab19-istio-token\") pod \"f616cbb0-e127-496b-be3c-55181e83ab19\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") "
Nov 02 17:47:32.804931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.804929    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-workload-certs\") pod \"f616cbb0-e127-496b-be3c-55181e83ab19\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") "
Nov 02 17:47:32.805195 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.804981    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f616cbb0-e127-496b-be3c-55181e83ab19-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f616cbb0-e127-496b-be3c-55181e83ab19-istio-podinfo\") pod \"f616cbb0-e127-496b-be3c-55181e83ab19\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") "
Nov 02 17:47:32.805195 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.805028    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-istio-envoy\") pod \"f616cbb0-e127-496b-be3c-55181e83ab19\" (UID: \"f616cbb0-e127-496b-be3c-55181e83ab19\") "
Nov 02 17:47:32.805195 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.805086    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-credential-socket" pod "f616cbb0-e127-496b-be3c-55181e83ab19" (UID: "f616cbb0-e127-496b-be3c-55181e83ab19"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:47:32.806094 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.805931    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-istio-data" pod "f616cbb0-e127-496b-be3c-55181e83ab19" (UID: "f616cbb0-e127-496b-be3c-55181e83ab19"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:47:32.806527 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.806348    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f616cbb0-e127-496b-be3c-55181e83ab19-istiod-ca-cert" pod "f616cbb0-e127-496b-be3c-55181e83ab19" (UID: "f616cbb0-e127-496b-be3c-55181e83ab19"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:47:32.806918 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.806654    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-workload-certs" pod "f616cbb0-e127-496b-be3c-55181e83ab19" (UID: "f616cbb0-e127-496b-be3c-55181e83ab19"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:47:32.810893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.808203    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-workload-socket" pod "f616cbb0-e127-496b-be3c-55181e83ab19" (UID: "f616cbb0-e127-496b-be3c-55181e83ab19"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:47:32.810893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.808513    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:32.810893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.808528    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:32.810893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.808538    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:32.810893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.808548    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f616cbb0-e127-496b-be3c-55181e83ab19-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:32.810893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.808556    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:32.816229 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.815995    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f616cbb0-e127-496b-be3c-55181e83ab19-kube-api-access-4z54v" pod "f616cbb0-e127-496b-be3c-55181e83ab19" (UID: "f616cbb0-e127-496b-be3c-55181e83ab19"). InnerVolumeSpecName "kube-api-access-4z54v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:47:32.816229 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.816077    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f616cbb0-e127-496b-be3c-55181e83ab19-istio-podinfo" pod "f616cbb0-e127-496b-be3c-55181e83ab19" (UID: "f616cbb0-e127-496b-be3c-55181e83ab19"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:47:32.816229 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.816143    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f616cbb0-e127-496b-be3c-55181e83ab19-istio-token" pod "f616cbb0-e127-496b-be3c-55181e83ab19" (UID: "f616cbb0-e127-496b-be3c-55181e83ab19"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:47:32.816229 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.816194    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-istio-envoy" pod "f616cbb0-e127-496b-be3c-55181e83ab19" (UID: "f616cbb0-e127-496b-be3c-55181e83ab19"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:47:32.840887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.840854    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:32.911589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.911549    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f616cbb0-e127-496b-be3c-55181e83ab19-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:32.911589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.911590    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f616cbb0-e127-496b-be3c-55181e83ab19-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:32.911909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.911607    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f616cbb0-e127-496b-be3c-55181e83ab19-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:32.911909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:32.911620    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4z54v\" (UniqueName: \"kubernetes.io/projected/f616cbb0-e127-496b-be3c-55181e83ab19-kube-api-access-4z54v\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:47:33.073894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.073744    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2" event={"ID":"20a26856-e0a4-410b-978b-4c65e143a80b","Type":"ContainerStarted","Data":"36d3f466742bc004aa1bcccf3f09c43559dd6325d3ef0d486dba3840c5ffaf38"}
Nov 02 17:47:33.073894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.073856    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2" event={"ID":"20a26856-e0a4-410b-978b-4c65e143a80b","Type":"ContainerStarted","Data":"2260b77e9d5b669a6eb507e90dbf5b42c7b883d214098d7687befbe44f307a0e"}
Nov 02 17:47:33.075504 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.075472    4456 generic.go:356] "Generic (PLEG): container finished" podID="f616cbb0-e127-496b-be3c-55181e83ab19" containerID="fa28955233cdf8a45b7a5132372edaddbffc9411d61ad23aeeb3bdd2eae2a2ac" exitCode=0
Nov 02 17:47:33.075698 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.075518    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"
Nov 02 17:47:33.076211 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.075517    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc" event={"ID":"f616cbb0-e127-496b-be3c-55181e83ab19","Type":"ContainerDied","Data":"fa28955233cdf8a45b7a5132372edaddbffc9411d61ad23aeeb3bdd2eae2a2ac"}
Nov 02 17:47:33.076349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.076216    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc" event={"ID":"f616cbb0-e127-496b-be3c-55181e83ab19","Type":"ContainerDied","Data":"41717dd259d2550394b183c5308c7cbf5adc1b8acc3cf8dce50eaf3a3d637d4e"}
Nov 02 17:47:33.076349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.076251    4456 scope.go:122] "RemoveContainer" containerID="fa28955233cdf8a45b7a5132372edaddbffc9411d61ad23aeeb3bdd2eae2a2ac"
Nov 02 17:47:33.122923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.122886    4456 scope.go:122] "RemoveContainer" containerID="fa28955233cdf8a45b7a5132372edaddbffc9411d61ad23aeeb3bdd2eae2a2ac"
Nov 02 17:47:33.123383 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:47:33.123354    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fa28955233cdf8a45b7a5132372edaddbffc9411d61ad23aeeb3bdd2eae2a2ac\": container with ID starting with fa28955233cdf8a45b7a5132372edaddbffc9411d61ad23aeeb3bdd2eae2a2ac not found: ID does not exist" containerID="fa28955233cdf8a45b7a5132372edaddbffc9411d61ad23aeeb3bdd2eae2a2ac"
Nov 02 17:47:33.123509 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.123399    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fa28955233cdf8a45b7a5132372edaddbffc9411d61ad23aeeb3bdd2eae2a2ac"} err="failed to get container status \"fa28955233cdf8a45b7a5132372edaddbffc9411d61ad23aeeb3bdd2eae2a2ac\": rpc error: code = NotFound desc = could not find container \"fa28955233cdf8a45b7a5132372edaddbffc9411d61ad23aeeb3bdd2eae2a2ac\": container with ID starting with fa28955233cdf8a45b7a5132372edaddbffc9411d61ad23aeeb3bdd2eae2a2ac not found: ID does not exist"
Nov 02 17:47:33.154659 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.154605    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2" podStartSLOduration=2.154588984 podStartE2EDuration="2.154588984s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:33.148587066 +0000 UTC m=+1069.639650229" watchObservedRunningTime="2026-11-02 17:47:33.154588984 +0000 UTC m=+1069.645652237"
Nov 02 17:47:33.212143 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.212097    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"]
Nov 02 17:47:33.217268 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.217233    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-derive-93321a4f-openshift-default-6d55476fb9-9t5bc"]
Nov 02 17:47:33.475054 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.475012    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:33.639842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.635282    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f616cbb0-e127-496b-be3c-55181e83ab19" path="/var/lib/kubelet/pods/f616cbb0-e127-496b-be3c-55181e83ab19/volumes"
Nov 02 17:47:33.661283 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.661250    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:33.750891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.746153    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:33.750891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.748194    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 02 17:47:33.750891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:33.748242    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2" podUID="20a26856-e0a4-410b-978b-4c65e143a80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 02 17:47:34.413575 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:34.413524    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:34.746028 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:34.745991    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 02 17:47:34.746198 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:34.746042    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2" podUID="20a26856-e0a4-410b-978b-4c65e143a80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 02 17:47:34.767619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:34.767572    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:35.745928 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:35.745895    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 02 17:47:35.746339 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:35.745947    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2" podUID="20a26856-e0a4-410b-978b-4c65e143a80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 02 17:47:36.650528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:36.650491    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:36.746263 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:36.745942    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 02 17:47:36.746263 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:36.746006    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2" podUID="20a26856-e0a4-410b-978b-4c65e143a80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 02 17:47:37.745939 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:37.745908    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused" start-of-body=
Nov 02 17:47:37.746123 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:37.745967    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2" podUID="20a26856-e0a4-410b-978b-4c65e143a80b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.123:15021/healthz/ready\": dial tcp 10.134.0.123:15021: connect: connection refused"
Nov 02 17:47:38.556770 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:38.556727    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:38.751211 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:38.751183    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:38.751594 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:38.751567    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:38.752256 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:38.752238    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:47:39.119803 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:39.119767    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"]
Nov 02 17:47:39.136227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:39.136195    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f0af0d7a-bks96"]
Nov 02 17:47:39.618430 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:39.618400    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="53281d37-399f-4063-a91e-b802ca02e1ff" path="/var/lib/kubelet/pods/53281d37-399f-4063-a91e-b802ca02e1ff/volumes"
Nov 02 17:47:40.546235 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:40.546202    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:41.047721 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:41.047693    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 17:47:42.283855 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:42.283785    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:44.044770 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.044734    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"]
Nov 02 17:47:44.045476 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.045461    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f616cbb0-e127-496b-be3c-55181e83ab19" containerName="istio-proxy"
Nov 02 17:47:44.045534 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.045478    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f616cbb0-e127-496b-be3c-55181e83ab19" containerName="istio-proxy"
Nov 02 17:47:44.045597 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.045589    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f616cbb0-e127-496b-be3c-55181e83ab19" containerName="istio-proxy"
Nov 02 17:47:44.051243 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.051217    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:44.073633 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.073601    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"]
Nov 02 17:47:44.077293 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.077267    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-504de7-serving-cert\""
Nov 02 17:47:44.113177 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.113138    4456 scope.go:122] "RemoveContainer" containerID="cce8ed0bab1398a692794e320e465b8efe5133255dfe647a72d88c2494357fce"
Nov 02 17:47:44.142529 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.142492    4456 scope.go:122] "RemoveContainer" containerID="ec233d01041de7bed6a6112b2c49a5fc01c5e6f2af99a81f226bb11368e94607"
Nov 02 17:47:44.153516 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.153472    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9051e10f-854f-4cc8-90a7-dda319c49c9a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:44.153625 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.153572    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9051e10f-854f-4cc8-90a7-dda319c49c9a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:44.153677 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.153639    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9051e10f-854f-4cc8-90a7-dda319c49c9a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:44.153977 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.153953    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n4fff\" (UniqueName: \"kubernetes.io/projected/9051e10f-854f-4cc8-90a7-dda319c49c9a-kube-api-access-n4fff\") pod \"maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:44.159415 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.159392    4456 scope.go:122] "RemoveContainer" containerID="58815a74d6cd4ebb7a553793ea8bb01fee47fc2d24f1ddd4222cf9349e10f1fd"
Nov 02 17:47:44.171314 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.171294    4456 scope.go:122] "RemoveContainer" containerID="2629f005a6d1a27d2d8f3d1800b2335b2c0713ddcadea2a0bc1196e58a3a324a"
Nov 02 17:47:44.178264 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.178240    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:44.183961 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.183935    4456 scope.go:122] "RemoveContainer" containerID="99d7fc8b557c0716e7c4eac696f3ab2813ca97ace85eac24a2568f311287af27"
Nov 02 17:47:44.194692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.194640    4456 scope.go:122] "RemoveContainer" containerID="c345096ba2dafb1e748d9c9b439faaf1c551d47f8e33d5de9d48d1e39cc3c87b"
Nov 02 17:47:44.256965 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.256924    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9051e10f-854f-4cc8-90a7-dda319c49c9a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:44.258107 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.256993    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9051e10f-854f-4cc8-90a7-dda319c49c9a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:44.258107 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.257081    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9051e10f-854f-4cc8-90a7-dda319c49c9a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:44.258107 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.257308    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n4fff\" (UniqueName: \"kubernetes.io/projected/9051e10f-854f-4cc8-90a7-dda319c49c9a-kube-api-access-n4fff\") pod \"maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:44.261223 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.261192    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9051e10f-854f-4cc8-90a7-dda319c49c9a-openshift-service-ca\") pod \"maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:44.263704 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.263662    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9051e10f-854f-4cc8-90a7-dda319c49c9a-maas-api-tls\") pod \"maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:44.276896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.271249    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9051e10f-854f-4cc8-90a7-dda319c49c9a-metrics-certs\") pod \"maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:44.300938 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.288781    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n4fff\" (UniqueName: \"kubernetes.io/projected/9051e10f-854f-4cc8-90a7-dda319c49c9a-kube-api-access-n4fff\") pod \"maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:44.359841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.358704    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:44.372841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.369455    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:44.731781 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:44.731739    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"]
Nov 02 17:47:44.731964 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:47:44.731800    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9051e10f_854f_4cc8_90a7_dda319c49c9a.slice/crio-69a4f9bb8dd22a44673b4a64d84589a7ab3203631cde91ea4d3f286018af26f9 WatchSource:0}: Error finding container 69a4f9bb8dd22a44673b4a64d84589a7ab3203631cde91ea4d3f286018af26f9: Status 404 returned error can't find the container with id 69a4f9bb8dd22a44673b4a64d84589a7ab3203631cde91ea4d3f286018af26f9
Nov 02 17:47:45.449883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:45.445052    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" event={"ID":"9051e10f-854f-4cc8-90a7-dda319c49c9a","Type":"ContainerStarted","Data":"69a4f9bb8dd22a44673b4a64d84589a7ab3203631cde91ea4d3f286018af26f9"}
Nov 02 17:47:46.175080 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:47:46.175039    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:47:46.412275 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:46.412224    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:46.488872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:46.485708    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" event={"ID":"9051e10f-854f-4cc8-90a7-dda319c49c9a","Type":"ContainerStarted","Data":"d337c60ba714489232e4f5ede5270e793cd4779290b761e8f42568d05e170851"}
Nov 02 17:47:46.488872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:46.485799    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:46.554353 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:46.553733    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podStartSLOduration=2.036027439 podStartE2EDuration="2.553713291s" totalImagesPullingTime="517.685852ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:47:44 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:47:46.550155241 +0000 UTC m=+1083.041218405" watchObservedRunningTime="2026-11-02 17:47:46.553713291 +0000 UTC m=+1083.044776455"
Nov 02 17:47:48.659882 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:48.655951    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:48.817469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:48.817384    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:49.486231 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:49.486191    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:50.671691 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:50.671654    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:52.520068 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:52.520036    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:52.546378 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:52.546355    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 17:47:52.675221 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:52.675162    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:54.475762 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:54.471493    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:56.274477 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:56.274446    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:58.099155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:58.099119    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:47:59.949308 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:47:59.949261    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:01.658646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:01.658591    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:03.427172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:03.427130    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:05.153514 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:05.153477    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:07.055046 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:07.055003    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:08.700993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:08.700947    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:09.047150 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:09.047057    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"]
Nov 02 17:48:09.051684 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:09.051656    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a86d703e-h8wd2"]
Nov 02 17:48:09.617985 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:09.617950    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="647ee669-772d-4e08-9cf2-73a204ff09fe" path="/var/lib/kubelet/pods/647ee669-772d-4e08-9cf2-73a204ff09fe/volumes"
Nov 02 17:48:10.548288 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:10.548252    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:11.552136 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:11.552102    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"]
Nov 02 17:48:11.558545 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:11.558522    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"
Nov 02 17:48:11.578863 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:11.576608    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"]
Nov 02 17:48:11.581920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:11.581897    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:48:11.582536 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:11.582512    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:48:11.595836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:11.595798    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dcfqd\" (UniqueName: \"kubernetes.io/projected/0af74da3-80ef-409f-a823-36fbb9b02d85-kube-api-access-dcfqd\") pod \"maas-api-revoke-sub-a4f1d499-jt8cp\" (UID: \"0af74da3-80ef-409f-a823-36fbb9b02d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"
Nov 02 17:48:11.596052 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:11.596032    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af74da3-80ef-409f-a823-36fbb9b02d85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4f1d499-jt8cp\" (UID: \"0af74da3-80ef-409f-a823-36fbb9b02d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"
Nov 02 17:48:11.596153 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:11.596070    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af74da3-80ef-409f-a823-36fbb9b02d85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4f1d499-jt8cp\" (UID: \"0af74da3-80ef-409f-a823-36fbb9b02d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"
Nov 02 17:48:11.698184 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:11.698141    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af74da3-80ef-409f-a823-36fbb9b02d85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4f1d499-jt8cp\" (UID: \"0af74da3-80ef-409f-a823-36fbb9b02d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"
Nov 02 17:48:11.698360 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:11.698199    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af74da3-80ef-409f-a823-36fbb9b02d85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4f1d499-jt8cp\" (UID: \"0af74da3-80ef-409f-a823-36fbb9b02d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"
Nov 02 17:48:11.698629 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:11.698611    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dcfqd\" (UniqueName: \"kubernetes.io/projected/0af74da3-80ef-409f-a823-36fbb9b02d85-kube-api-access-dcfqd\") pod \"maas-api-revoke-sub-a4f1d499-jt8cp\" (UID: \"0af74da3-80ef-409f-a823-36fbb9b02d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"
Nov 02 17:48:11.698935 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:11.698912    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af74da3-80ef-409f-a823-36fbb9b02d85-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4f1d499-jt8cp\" (UID: \"0af74da3-80ef-409f-a823-36fbb9b02d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"
Nov 02 17:48:11.701404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:11.701383    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af74da3-80ef-409f-a823-36fbb9b02d85-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4f1d499-jt8cp\" (UID: \"0af74da3-80ef-409f-a823-36fbb9b02d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"
Nov 02 17:48:11.723752 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:11.723720    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dcfqd\" (UniqueName: \"kubernetes.io/projected/0af74da3-80ef-409f-a823-36fbb9b02d85-kube-api-access-dcfqd\") pod \"maas-api-revoke-sub-a4f1d499-jt8cp\" (UID: \"0af74da3-80ef-409f-a823-36fbb9b02d85\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"
Nov 02 17:48:11.880140 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:11.880055    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"
Nov 02 17:48:12.154892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:12.141672    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:12.262476 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:12.262325    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"]
Nov 02 17:48:12.274609 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:48:12.274353    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0af74da3_80ef_409f_a823_36fbb9b02d85.slice/crio-e12cd4d6a96d9a4b5dde94ab791ae43095c25da0f38a4e80dc3a7c2ac3a6dafc WatchSource:0}: Error finding container e12cd4d6a96d9a4b5dde94ab791ae43095c25da0f38a4e80dc3a7c2ac3a6dafc: Status 404 returned error can't find the container with id e12cd4d6a96d9a4b5dde94ab791ae43095c25da0f38a4e80dc3a7c2ac3a6dafc
Nov 02 17:48:12.329535 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:12.329503    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:12.598874 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:12.598792    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:12.909655 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:12.909551    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp" event={"ID":"0af74da3-80ef-409f-a823-36fbb9b02d85","Type":"ContainerStarted","Data":"e753de84f69349c10975edd00c14c84deea4e4b9c17dc64bec1403582cb06593"}
Nov 02 17:48:12.909655 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:12.909614    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp" event={"ID":"0af74da3-80ef-409f-a823-36fbb9b02d85","Type":"ContainerStarted","Data":"e12cd4d6a96d9a4b5dde94ab791ae43095c25da0f38a4e80dc3a7c2ac3a6dafc"}
Nov 02 17:48:12.955787 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:12.955732    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp" podStartSLOduration=1.955718637 podStartE2EDuration="1.955718637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:11 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:12.952865253 +0000 UTC m=+1109.443928429" watchObservedRunningTime="2026-11-02 17:48:12.955718637 +0000 UTC m=+1109.446781798"
Nov 02 17:48:13.034372 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.034335    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"]
Nov 02 17:48:13.040642 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.040616    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.061750 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.061713    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-c810ca-openshift-default-dockercfg-s9n8v\""
Nov 02 17:48:13.063156 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.063128    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"]
Nov 02 17:48:13.125739 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.125703    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:13.179449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.177292    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-data\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.179449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.177441    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-workload-socket\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.179449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.177473    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d6rw4\" (UniqueName: \"kubernetes.io/projected/2dc48441-ccc0-47fc-abe7-7c327559eba4-kube-api-access-d6rw4\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.179449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.177649    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-credential-socket\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.179449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.178004    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-workload-certs\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.179449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.178110    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-token\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.179449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.178222    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc48441-ccc0-47fc-abe7-7c327559eba4-istiod-ca-cert\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.179449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.178258    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-envoy\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.179449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.178305    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-podinfo\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.286839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.280151    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-token\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.286839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.280285    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc48441-ccc0-47fc-abe7-7c327559eba4-istiod-ca-cert\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.286839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.280320    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-envoy\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.286839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.280354    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-podinfo\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.286839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.280505    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-data\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.286839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.280965    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-workload-socket\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.286839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.280993    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d6rw4\" (UniqueName: \"kubernetes.io/projected/2dc48441-ccc0-47fc-abe7-7c327559eba4-kube-api-access-d6rw4\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.286839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.281153    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-credential-socket\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.286839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.281176    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-workload-certs\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.286839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.281547    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-workload-certs\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.286839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.281828    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-data\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.286839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.281885    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc48441-ccc0-47fc-abe7-7c327559eba4-istiod-ca-cert\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.286839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.282066    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-workload-socket\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.286839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.282276    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-credential-socket\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.290432 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.290402    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-envoy\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.319954 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.319920    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-token\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.320997 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.320967    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d6rw4\" (UniqueName: \"kubernetes.io/projected/2dc48441-ccc0-47fc-abe7-7c327559eba4-kube-api-access-d6rw4\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.331870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.329939    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:13.331870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.331131    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-podinfo\") pod \"e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") " pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.360874 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.360428    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:13.858348 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:48:13.858314    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2dc48441_ccc0_47fc_abe7_7c327559eba4.slice/crio-0b3d9d3727e971980971b177f1eac5f801f346dfcc99842d0439d668b81f28ad WatchSource:0}: Error finding container 0b3d9d3727e971980971b177f1eac5f801f346dfcc99842d0439d668b81f28ad: Status 404 returned error can't find the container with id 0b3d9d3727e971980971b177f1eac5f801f346dfcc99842d0439d668b81f28ad
Nov 02 17:48:13.859359 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.859320    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"]
Nov 02 17:48:13.861018 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.860987    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:13.861123 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.861052    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:13.861123 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.861080    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:13.921578 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.921531    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" event={"ID":"2dc48441-ccc0-47fc-abe7-7c327559eba4","Type":"ContainerStarted","Data":"0b3d9d3727e971980971b177f1eac5f801f346dfcc99842d0439d668b81f28ad"}
Nov 02 17:48:13.923732 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.923692    4456 generic.go:356] "Generic (PLEG): container finished" podID="0af74da3-80ef-409f-a823-36fbb9b02d85" containerID="e753de84f69349c10975edd00c14c84deea4e4b9c17dc64bec1403582cb06593" exitCode=0
Nov 02 17:48:13.923944 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:13.923915    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp" event={"ID":"0af74da3-80ef-409f-a823-36fbb9b02d85","Type":"ContainerDied","Data":"e753de84f69349c10975edd00c14c84deea4e4b9c17dc64bec1403582cb06593"}
Nov 02 17:48:14.328720 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:14.328665    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:14.847065 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:14.846997    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:14.940950 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:14.940906    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" event={"ID":"2dc48441-ccc0-47fc-abe7-7c327559eba4","Type":"ContainerStarted","Data":"1a22dd302b6925abeb1cc285dfdf5df0473f58e4b460c5dab034c0a14fa18275"}
Nov 02 17:48:15.141358 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:15.140547    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"
Nov 02 17:48:15.147061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:15.147029    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0af74da3-80ef-409f-a823-36fbb9b02d85-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af74da3-80ef-409f-a823-36fbb9b02d85-maas-api-xks-ca\") pod \"0af74da3-80ef-409f-a823-36fbb9b02d85\" (UID: \"0af74da3-80ef-409f-a823-36fbb9b02d85\") "
Nov 02 17:48:15.147236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:15.147124    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0af74da3-80ef-409f-a823-36fbb9b02d85-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af74da3-80ef-409f-a823-36fbb9b02d85-maas-api-openshift-ca\") pod \"0af74da3-80ef-409f-a823-36fbb9b02d85\" (UID: \"0af74da3-80ef-409f-a823-36fbb9b02d85\") "
Nov 02 17:48:15.147236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:15.147175    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0af74da3-80ef-409f-a823-36fbb9b02d85-kube-api-access-dcfqd\" (UniqueName: \"kubernetes.io/projected/0af74da3-80ef-409f-a823-36fbb9b02d85-kube-api-access-dcfqd\") pod \"0af74da3-80ef-409f-a823-36fbb9b02d85\" (UID: \"0af74da3-80ef-409f-a823-36fbb9b02d85\") "
Nov 02 17:48:15.147520 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:15.147488    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0af74da3-80ef-409f-a823-36fbb9b02d85-maas-api-openshift-ca" pod "0af74da3-80ef-409f-a823-36fbb9b02d85" (UID: "0af74da3-80ef-409f-a823-36fbb9b02d85"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:48:15.147987 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:15.147968    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/0af74da3-80ef-409f-a823-36fbb9b02d85-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:48:15.149533 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:15.149509    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0af74da3-80ef-409f-a823-36fbb9b02d85-maas-api-xks-ca" pod "0af74da3-80ef-409f-a823-36fbb9b02d85" (UID: "0af74da3-80ef-409f-a823-36fbb9b02d85"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:48:15.149623 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:15.149564    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0af74da3-80ef-409f-a823-36fbb9b02d85-kube-api-access-dcfqd" pod "0af74da3-80ef-409f-a823-36fbb9b02d85" (UID: "0af74da3-80ef-409f-a823-36fbb9b02d85"). InnerVolumeSpecName "kube-api-access-dcfqd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:48:15.193950 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:15.193895    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" podStartSLOduration=3.193878672 podStartE2EDuration="3.193878672s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:15.011672942 +0000 UTC m=+1111.502736106" watchObservedRunningTime="2026-11-02 17:48:15.193878672 +0000 UTC m=+1111.684941833"
Nov 02 17:48:15.259836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:15.259333    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/0af74da3-80ef-409f-a823-36fbb9b02d85-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:48:15.259836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:15.259375    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dcfqd\" (UniqueName: \"kubernetes.io/projected/0af74da3-80ef-409f-a823-36fbb9b02d85-kube-api-access-dcfqd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:48:15.360945 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:15.360911    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:15.366782 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:15.366758    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:16.100498 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:16.100458    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:16.101014 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:16.100673    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"
Nov 02 17:48:16.110539 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:16.110090    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp" event={"ID":"0af74da3-80ef-409f-a823-36fbb9b02d85","Type":"ContainerDied","Data":"e12cd4d6a96d9a4b5dde94ab791ae43095c25da0f38a4e80dc3a7c2ac3a6dafc"}
Nov 02 17:48:16.110539 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:16.110186    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e12cd4d6a96d9a4b5dde94ab791ae43095c25da0f38a4e80dc3a7c2ac3a6dafc"
Nov 02 17:48:16.110946 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:16.110678    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:16.113163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:16.113136    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:48:16.176110 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:48:16.176067    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:48:17.003380 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:17.003342    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:17.696841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:17.696310    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:17.892659 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:17.892628    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"]
Nov 02 17:48:17.893628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:17.893600    4456 cpu_manager.go:414] "containerMap: removing container" podUID="0af74da3-80ef-409f-a823-36fbb9b02d85" containerName="revoke-keys"
Nov 02 17:48:17.893628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:17.893627    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0af74da3-80ef-409f-a823-36fbb9b02d85" containerName="revoke-keys"
Nov 02 17:48:17.894212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:17.893896    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="0af74da3-80ef-409f-a823-36fbb9b02d85" containerName="revoke-keys"
Nov 02 17:48:17.898252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:17.898228    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:17.926688 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:17.926656    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"]
Nov 02 17:48:17.929349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:17.929324    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-04e682fe-openshift-default-dockercfg-2whl8\""
Nov 02 17:48:18.005487 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.005389    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-data\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.005651 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.005510    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-credential-socket\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.005651 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.005562    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istiod-ca-cert\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.005651 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.005609    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-workload-socket\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.005870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.005792    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-workload-certs\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.005946 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.005865    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-podinfo\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.006006 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.005988    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ztztx\" (UniqueName: \"kubernetes.io/projected/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-kube-api-access-ztztx\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.006087 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.006073    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-token\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.006163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.006141    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-envoy\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.007186 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.007163    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:18.119621 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.119579    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-data\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.119959 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.119935    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-credential-socket\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.120124 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.120108    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istiod-ca-cert\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.120277 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.120263    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-workload-socket\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.120471 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.120457    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-workload-certs\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.120597 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.120585    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-podinfo\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.120775 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.120759    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ztztx\" (UniqueName: \"kubernetes.io/projected/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-kube-api-access-ztztx\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.120979 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.120959    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-token\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.121106 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.121091    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-envoy\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.122203 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.122179    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-workload-socket\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.122654 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.122634    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-data\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.128889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.124134    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-credential-socket\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.129083 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.124724    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istiod-ca-cert\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.129308 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.124984    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-workload-certs\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.145280 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.141068    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-podinfo\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.147155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.147127    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-envoy\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.153523 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.153494    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ztztx\" (UniqueName: \"kubernetes.io/projected/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-kube-api-access-ztztx\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.156270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.156231    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-token\") pod \"e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") " pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.179054 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.179020    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:18.214507 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.214235    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:18.354913 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.354875    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:18.593932 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.593846    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"]
Nov 02 17:48:18.598916 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:48:18.598884    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod0e46fada_1bd0_4b22_96ab_3c49c4b3199e.slice/crio-82d5c328a8a272ae602806e4c4454bc5b032f44d31e955b664f22927e524a38b WatchSource:0}: Error finding container 82d5c328a8a272ae602806e4c4454bc5b032f44d31e955b664f22927e524a38b: Status 404 returned error can't find the container with id 82d5c328a8a272ae602806e4c4454bc5b032f44d31e955b664f22927e524a38b
Nov 02 17:48:18.602360 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.602322    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:18.602447 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.602412    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:18.602494 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:18.602452    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:19.168308 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:19.168267    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6" event={"ID":"0e46fada-1bd0-4b22-96ab-3c49c4b3199e","Type":"ContainerStarted","Data":"131e4bce0b28eec82743c5f5c4e6857572a311fdf264f6220cebc081de88ca3e"}
Nov 02 17:48:19.168697 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:19.168359    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6" event={"ID":"0e46fada-1bd0-4b22-96ab-3c49c4b3199e","Type":"ContainerStarted","Data":"82d5c328a8a272ae602806e4c4454bc5b032f44d31e955b664f22927e524a38b"}
Nov 02 17:48:19.214528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:19.214489    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:19.222208 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:19.222182    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:19.222390 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:19.222229    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6" podStartSLOduration=2.222216167 podStartE2EDuration="2.222216167s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:19.217565744 +0000 UTC m=+1115.708628928" watchObservedRunningTime="2026-11-02 17:48:19.222216167 +0000 UTC m=+1115.713279328"
Nov 02 17:48:19.505079 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:19.505035    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:19.901931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:19.847048    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:20.051578 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:20.051537    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"]
Nov 02 17:48:20.062298 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:20.062250    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-887fd0a8-nxk57"]
Nov 02 17:48:20.177235 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:20.177150    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:20.179552 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:20.179531    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:48:20.393346 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:20.390195    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:20.862311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:20.862261    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:21.347653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:21.347603    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:21.624889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:21.624307    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="64c925ba-deb0-4b7f-9542-7ffd5719576d" path="/var/lib/kubelet/pods/64c925ba-deb0-4b7f-9542-7ffd5719576d/volumes"
Nov 02 17:48:22.104602 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:22.104561    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:22.295622 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:22.295584    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:22.517836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:22.504937    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:24.027353 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:24.027309    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:24.163725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:24.163681    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"]
Nov 02 17:48:24.170916 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:24.170890    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"
Nov 02 17:48:24.199901 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:24.199870    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-04e682fe-dockercfg-f7nb2\""
Nov 02 17:48:24.262413 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:24.260712    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-04e682fe\""
Nov 02 17:48:24.265926 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:24.265421    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:24.269865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:24.269841    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"]
Nov 02 17:48:24.377174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:24.377067    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4nxcw\" (UniqueName: \"kubernetes.io/projected/839ebaa1-25e5-4db7-abdc-7f55f837bff7-kube-api-access-4nxcw\") pod \"payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb\" (UID: \"839ebaa1-25e5-4db7-abdc-7f55f837bff7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"
Nov 02 17:48:24.387921 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:24.377506    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839ebaa1-25e5-4db7-abdc-7f55f837bff7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb\" (UID: \"839ebaa1-25e5-4db7-abdc-7f55f837bff7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"
Nov 02 17:48:24.548088 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:24.543054    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4nxcw\" (UniqueName: \"kubernetes.io/projected/839ebaa1-25e5-4db7-abdc-7f55f837bff7-kube-api-access-4nxcw\") pod \"payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb\" (UID: \"839ebaa1-25e5-4db7-abdc-7f55f837bff7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"
Nov 02 17:48:24.548088 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:24.543183    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839ebaa1-25e5-4db7-abdc-7f55f837bff7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb\" (UID: \"839ebaa1-25e5-4db7-abdc-7f55f837bff7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"
Nov 02 17:48:24.548088 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:24.546286    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839ebaa1-25e5-4db7-abdc-7f55f837bff7-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb\" (UID: \"839ebaa1-25e5-4db7-abdc-7f55f837bff7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"
Nov 02 17:48:24.640676 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:24.640599    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4nxcw\" (UniqueName: \"kubernetes.io/projected/839ebaa1-25e5-4db7-abdc-7f55f837bff7-kube-api-access-4nxcw\") pod \"payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb\" (UID: \"839ebaa1-25e5-4db7-abdc-7f55f837bff7\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"
Nov 02 17:48:24.741034 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:24.739410    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"
Nov 02 17:48:24.875797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:24.875647    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:25.143142 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:25.143108    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"]
Nov 02 17:48:25.321870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:25.320711    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb" event={"ID":"839ebaa1-25e5-4db7-abdc-7f55f837bff7","Type":"ContainerStarted","Data":"648f137883d0ad44adb7ec5c721265dd4b4d1dda41c2a3489d6187de719dab72"}
Nov 02 17:48:25.321870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:25.320789    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb" event={"ID":"839ebaa1-25e5-4db7-abdc-7f55f837bff7","Type":"ContainerStarted","Data":"7bc801eec8efb883a7de656646f5f5034a5b335b228f1292da67979c073c5f6d"}
Nov 02 17:48:25.321870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:25.321649    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"
Nov 02 17:48:25.369885 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:25.369832    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb" podStartSLOduration=1.3698033920000001 podStartE2EDuration="1.369803392s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:25.365049565 +0000 UTC m=+1121.856112724" watchObservedRunningTime="2026-11-02 17:48:25.369803392 +0000 UTC m=+1121.860866553"
Nov 02 17:48:26.666562 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:26.666533    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:26.831346 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:26.831305    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:28.517506 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:28.517466    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:28.670352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:28.670312    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:28.737908 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:28.737873    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"]
Nov 02 17:48:28.746137 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:28.746108    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"
Nov 02 17:48:28.763024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:28.762996    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"]
Nov 02 17:48:28.789938 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:28.772267    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c4dbe3\""
Nov 02 17:48:28.789938 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:28.777796    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c4dbe3-dockercfg-qp9g8\""
Nov 02 17:48:28.850768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:28.850173    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8847684-5676-4253-850f-9f2b2c42ea0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg\" (UID: \"b8847684-5676-4253-850f-9f2b2c42ea0c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"
Nov 02 17:48:28.850768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:28.850314    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-psdvm\" (UniqueName: \"kubernetes.io/projected/b8847684-5676-4253-850f-9f2b2c42ea0c-kube-api-access-psdvm\") pod \"payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg\" (UID: \"b8847684-5676-4253-850f-9f2b2c42ea0c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"
Nov 02 17:48:28.862473 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:28.862368    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:28.965991 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:28.965955    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8847684-5676-4253-850f-9f2b2c42ea0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg\" (UID: \"b8847684-5676-4253-850f-9f2b2c42ea0c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"
Nov 02 17:48:28.970842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:28.966383    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-psdvm\" (UniqueName: \"kubernetes.io/projected/b8847684-5676-4253-850f-9f2b2c42ea0c-kube-api-access-psdvm\") pod \"payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg\" (UID: \"b8847684-5676-4253-850f-9f2b2c42ea0c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"
Nov 02 17:48:28.995842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:28.972642    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8847684-5676-4253-850f-9f2b2c42ea0c-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg\" (UID: \"b8847684-5676-4253-850f-9f2b2c42ea0c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"
Nov 02 17:48:29.160528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:29.155331    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-psdvm\" (UniqueName: \"kubernetes.io/projected/b8847684-5676-4253-850f-9f2b2c42ea0c-kube-api-access-psdvm\") pod \"payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg\" (UID: \"b8847684-5676-4253-850f-9f2b2c42ea0c\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"
Nov 02 17:48:29.452326 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:29.452293    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"
Nov 02 17:48:29.806845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:29.806744    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"]
Nov 02 17:48:30.387560 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:30.387518    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" event={"ID":"b8847684-5676-4253-850f-9f2b2c42ea0c","Type":"ContainerStarted","Data":"1a5f4c3158352df507219d936b2b2df3545243e213952a4a8b0fa5bfff6d9c28"}
Nov 02 17:48:30.387749 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:30.387629    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"
Nov 02 17:48:30.387749 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:30.387656    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" event={"ID":"b8847684-5676-4253-850f-9f2b2c42ea0c","Type":"ContainerStarted","Data":"0931e1faf1d8a7f8328258d16e252e3b301ac6a4b125a3570dc2b7a0f74640d5"}
Nov 02 17:48:30.435116 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:30.435053    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" podStartSLOduration=2.435037978 podStartE2EDuration="2.435037978s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:28 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:30.432480554 +0000 UTC m=+1126.923543751" watchObservedRunningTime="2026-11-02 17:48:30.435037978 +0000 UTC m=+1126.926101147"
Nov 02 17:48:30.747223 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:30.747189    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:30.900788 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:30.900689    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:31.135884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:31.135769    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"]
Nov 02 17:48:31.148169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:31.148138    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-81dc9fb1-wqtt9"]
Nov 02 17:48:31.620566 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:31.620516    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b172da9-d254-45eb-aa9f-16e72704dd06" path="/var/lib/kubelet/pods/9b172da9-d254-45eb-aa9f-16e72704dd06/volumes"
Nov 02 17:48:32.508251 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:32.508211    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:34.254049 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:34.253996    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:34.537152 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:34.537042    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:36.099579 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.099483    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:36.746409 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.745767    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"]
Nov 02 17:48:36.753272 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.753244    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.782527 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.782480    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"]
Nov 02 17:48:36.873506 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.873471    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-credential-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.873506 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.873506    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-envoy\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.873764 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.873529    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istiod-ca-cert\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.873764 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.873742    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-data\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.873987 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.873872    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-podinfo\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.874316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.873993    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-workload-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.874316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.874103    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qrgxc\" (UniqueName: \"kubernetes.io/projected/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-kube-api-access-qrgxc\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.874316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.874249    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-token\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.874316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.874296    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-workload-certs\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.976529 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.976442    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-workload-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.976707 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.976549    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qrgxc\" (UniqueName: \"kubernetes.io/projected/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-kube-api-access-qrgxc\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.976707 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.976688    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-token\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.976947 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.976720    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-workload-certs\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.976947 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.976855    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-credential-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.976947 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.976875    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-workload-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.976947 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.976881    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-envoy\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.977171 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.976949    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istiod-ca-cert\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.977171 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.977055    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-data\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.977171 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.977117    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-workload-certs\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.977171 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.977164    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-podinfo\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.977391 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.977271    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-credential-socket\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.977455 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.977394    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-data\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.977771 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.977748    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istiod-ca-cert\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.988656 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.988630    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-envoy\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:36.989184 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:36.989163    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-podinfo\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:37.006026 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:37.005998    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-token\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:37.006387 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:37.006367    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qrgxc\" (UniqueName: \"kubernetes.io/projected/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-kube-api-access-qrgxc\") pod \"e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") " pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:37.092234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:37.092202    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:37.358204 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:37.358134    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"
Nov 02 17:48:37.868379 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:37.868342    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"]
Nov 02 17:48:37.912601 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:37.912560    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:37.945443 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:37.945399    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:37.945590 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:37.945484    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:37.973207 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:48:37.936833    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5f82179e_cc9c_4899_a6fc_1a75c4d5d517.slice/crio-5d425e10bddf403e64054e56f00f71506fb794b6013c884fae5786249b1a1530 WatchSource:0}: Error finding container 5d425e10bddf403e64054e56f00f71506fb794b6013c884fae5786249b1a1530: Status 404 returned error can't find the container with id 5d425e10bddf403e64054e56f00f71506fb794b6013c884fae5786249b1a1530
Nov 02 17:48:38.082412 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:38.082370    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:38.474841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:38.474186    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm" event={"ID":"5f82179e-cc9c-4899-a6fc-1a75c4d5d517","Type":"ContainerStarted","Data":"b07092894c873f382807fe54f492a1c0a94572b17d28096d4fe738a720dc75c0"}
Nov 02 17:48:38.474841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:38.474775    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm" event={"ID":"5f82179e-cc9c-4899-a6fc-1a75c4d5d517","Type":"ContainerStarted","Data":"5d425e10bddf403e64054e56f00f71506fb794b6013c884fae5786249b1a1530"}
Nov 02 17:48:38.537863 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:38.537583    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm" podStartSLOduration=2.5375391819999997 podStartE2EDuration="2.537539182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:38.527593168 +0000 UTC m=+1135.018656376" watchObservedRunningTime="2026-11-02 17:48:38.537539182 +0000 UTC m=+1135.028602344"
Nov 02 17:48:39.102845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:39.093321    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:39.105573 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:39.105208    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 17:48:39.105573 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:39.105273    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm" podUID="5f82179e-cc9c-4899-a6fc-1a75c4d5d517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 17:48:40.097665 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:40.097298    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 17:48:40.097665 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:40.097363    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm" podUID="5f82179e-cc9c-4899-a6fc-1a75c4d5d517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 17:48:40.113848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:40.108078    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:41.093791 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:41.093755    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 17:48:41.094115 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:41.094083    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm" podUID="5f82179e-cc9c-4899-a6fc-1a75c4d5d517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 17:48:41.410891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:41.410089    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"
Nov 02 17:48:41.954424 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:41.954344    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:42.005852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:42.004942    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:48:42.005852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:42.005010    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt" podUID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:48:42.114844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:42.109604    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 17:48:42.114844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:42.109665    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm" podUID="5f82179e-cc9c-4899-a6fc-1a75c4d5d517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 17:48:42.130840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:42.121328    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:42.615127 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:42.615092    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:43.113841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:43.113254    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 17:48:43.113841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:43.113318    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm" podUID="5f82179e-cc9c-4899-a6fc-1a75c4d5d517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 17:48:43.946886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:43.946837    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:44.060932 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:44.060851    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"]
Nov 02 17:48:44.071613 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:44.070776    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2585040a-p8z5j"]
Nov 02 17:48:44.097858 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:44.092705    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused" start-of-body=
Nov 02 17:48:44.097858 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:44.092768    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm" podUID="5f82179e-cc9c-4899-a6fc-1a75c4d5d517" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.130:15021/healthz/ready\": dial tcp 10.134.0.130:15021: connect: connection refused"
Nov 02 17:48:44.348993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:44.348906    4456 scope.go:122] "RemoveContainer" containerID="0dfa4b4d3d083585f98bf2bea0a07a716bca677f6de2e9b57f1682e3548cadde"
Nov 02 17:48:44.628724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:44.628637    4456 generic.go:356] "Generic (PLEG): container finished" podID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerID="6f61a792111d21e8b0f4badd27b2c7c04693431f7923f8a8a2d0449f05484e9e" exitCode=0
Nov 02 17:48:44.628724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:44.628704    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt" event={"ID":"627671f3-ee4e-4ff9-b03f-235489bc356f","Type":"ContainerDied","Data":"6f61a792111d21e8b0f4badd27b2c7c04693431f7923f8a8a2d0449f05484e9e"}
Nov 02 17:48:44.629275 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:44.629252    4456 scope.go:122] "RemoveContainer" containerID="6f61a792111d21e8b0f4badd27b2c7c04693431f7923f8a8a2d0449f05484e9e"
Nov 02 17:48:44.631271 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:44.631242    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:44.631358 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:44.631324    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:44.631397 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:44.631370    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:45.098083 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:45.098051    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:45.098658 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:45.098629    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:45.099424 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:45.099403    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:48:45.646677 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:45.646642    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f59320ce-836a-422e-9717-61df0ec341d0" path="/var/lib/kubelet/pods/f59320ce-836a-422e-9717-61df0ec341d0/volumes"
Nov 02 17:48:45.647298 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:45.647272    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt" event={"ID":"627671f3-ee4e-4ff9-b03f-235489bc356f","Type":"ContainerStarted","Data":"e9384c1e62fcac2aa6d54f5deceeef3155733458debf08d0a878bd911b7b27de"}
Nov 02 17:48:45.816764 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:45.816715    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:46.072347 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:46.072318    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:48:46.072558 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:46.072539    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 17:48:46.072609 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:46.072583    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt" podUID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 17:48:46.227269 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:48:46.227232    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:48:47.072334 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:47.072296    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 17:48:47.072543 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:47.072354    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt" podUID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 17:48:47.748640 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:47.748556    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:47.904842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:47.903887    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:48.071913 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:48.071830    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 17:48:48.071913 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:48.071893    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt" podUID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 17:48:49.074844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:49.074330    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 17:48:49.074844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:49.074432    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt" podUID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 17:48:49.745162 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:49.745121    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:50.072034 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:50.071945    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused" start-of-body=
Nov 02 17:48:50.072034 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:50.071999    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt" podUID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.121:15021/healthz/ready\": dial tcp 10.134.0.121:15021: connect: connection refused"
Nov 02 17:48:51.076102 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:51.076068    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:48:51.076560 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:51.076468    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:48:51.077214 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:51.077194    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 17:48:51.403609 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:51.403520    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:52.488258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:52.488229    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:52.889650 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:52.889568    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"]
Nov 02 17:48:52.893845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:52.893806    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:52.906084 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:52.906056    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-token\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:52.906209 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:52.906113    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istiod-ca-cert\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:52.906252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:52.906205    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-workload-socket\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:52.906252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:52.906234    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w6dvp\" (UniqueName: \"kubernetes.io/projected/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-kube-api-access-w6dvp\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:52.906345 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:52.906281    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-workload-certs\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:52.906410 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:52.906385    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-data\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:52.906478 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:52.906459    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-podinfo\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:52.906629 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:52.906614    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-credential-socket\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:52.906730 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:52.906715    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-envoy\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:52.921130 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:52.921100    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"]
Nov 02 17:48:52.933257 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:52.933220    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-mt-0d1f5d84-openshift-default-dockercfg-drwgg\""
Nov 02 17:48:52.950550 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:52.950520    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:53.063339 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.063268    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-data\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.063804 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.063781    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-data\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.064029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.064005    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-podinfo\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.064258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.064238    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-credential-socket\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.064898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.064877    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-envoy\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.065246 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.065223    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-token\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.065476 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.065457    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istiod-ca-cert\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.065604 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.065587    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-workload-socket\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.065705 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.065691    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w6dvp\" (UniqueName: \"kubernetes.io/projected/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-kube-api-access-w6dvp\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.065917 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.065899    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-workload-certs\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.066982 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.066570    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-credential-socket\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.067777 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.067736    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istiod-ca-cert\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.068422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.068357    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-workload-socket\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.068611 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.068593    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-workload-certs\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.130838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.129233    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-envoy\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.130838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.129753    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-podinfo\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.175108 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.171124    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-token\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.175108 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.171743    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w6dvp\" (UniqueName: \"kubernetes.io/projected/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-kube-api-access-w6dvp\") pod \"e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") " pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.191511 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.191453    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:53.224842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.218285    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:53.357585 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.356791    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:53.571511 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.571475    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:53.721597 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:48:53.721558    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2268303a_b8a6_4783_b4fc_3467cd0c2bb3.slice/crio-e203f4a74424024d2da5a0ce673ce75ed761416ad9269f3e80872b94f388bfe5 WatchSource:0}: Error finding container e203f4a74424024d2da5a0ce673ce75ed761416ad9269f3e80872b94f388bfe5: Status 404 returned error can't find the container with id e203f4a74424024d2da5a0ce673ce75ed761416ad9269f3e80872b94f388bfe5
Nov 02 17:48:53.725800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.725771    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"]
Nov 02 17:48:53.731736 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.731679    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:53.731917 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.731779    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:53.731917 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.731869    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:48:53.839153 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:53.839070    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4" event={"ID":"2268303a-b8a6-4783-b4fc-3467cd0c2bb3","Type":"ContainerStarted","Data":"e203f4a74424024d2da5a0ce673ce75ed761416ad9269f3e80872b94f388bfe5"}
Nov 02 17:48:54.042381 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:54.042238    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"]
Nov 02 17:48:54.064838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:54.064640    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-51e76e64-rzt6w"]
Nov 02 17:48:54.850248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:54.850211    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4" event={"ID":"2268303a-b8a6-4783-b4fc-3467cd0c2bb3","Type":"ContainerStarted","Data":"32c5e815dd3a88821539f897b9d64cab531b39e5121bc42aa66691c79ec0ae1b"}
Nov 02 17:48:54.912654 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:54.912600    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4" podStartSLOduration=2.9125854159999998 podStartE2EDuration="2.912585416s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:48:54.910382644 +0000 UTC m=+1151.401445808" watchObservedRunningTime="2026-11-02 17:48:54.912585416 +0000 UTC m=+1151.403648577"
Nov 02 17:48:55.219172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:55.219130    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:55.225842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:55.225796    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:55.270558 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:55.270034    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:55.525746 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:55.525650    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:55.618643 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:55.618609    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e86b922b-7bdc-4b34-ba94-cf10ffa14ed2" path="/var/lib/kubelet/pods/e86b922b-7bdc-4b34-ba94-cf10ffa14ed2/volumes"
Nov 02 17:48:55.860750 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:55.860668    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:55.861663 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:55.861644    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:48:56.076833 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:56.076784    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:56.429851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:56.425832    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:57.354839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:57.354754    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:57.650077 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:57.649253    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:57.815157 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:57.815117    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:59.092791 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:59.092734    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"]
Nov 02 17:48:59.102038 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:59.101987    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4c36e96c-g2gzw"]
Nov 02 17:48:59.683714 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:59.683683    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:59.791698 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:59.791658    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99" path="/var/lib/kubelet/pods/2ac958e4-e1dc-45a6-94e5-d9f2b95e0a99/volumes"
Nov 02 17:48:59.914228 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:59.914190    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:48:59.949958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:59.949868    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"]
Nov 02 17:48:59.978841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:48:59.960272    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"]
Nov 02 17:49:00.030693 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.013775    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"
Nov 02 17:49:00.087898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.087867    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-0d1f5d84-dockercfg-dp6hc\""
Nov 02 17:49:00.096187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.096160    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-0d1f5d84\""
Nov 02 17:49:00.165962 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.132694    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hck44\" (UniqueName: \"kubernetes.io/projected/2612d124-cc29-46af-bad1-f8af29bee00f-kube-api-access-hck44\") pod \"payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr\" (UID: \"2612d124-cc29-46af-bad1-f8af29bee00f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"
Nov 02 17:49:00.165962 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.132787    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2612d124-cc29-46af-bad1-f8af29bee00f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr\" (UID: \"2612d124-cc29-46af-bad1-f8af29bee00f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"
Nov 02 17:49:00.246984 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.206635    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"]
Nov 02 17:49:00.287859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.249660    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hck44\" (UniqueName: \"kubernetes.io/projected/2612d124-cc29-46af-bad1-f8af29bee00f-kube-api-access-hck44\") pod \"payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr\" (UID: \"2612d124-cc29-46af-bad1-f8af29bee00f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"
Nov 02 17:49:00.287859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.249754    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2612d124-cc29-46af-bad1-f8af29bee00f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr\" (UID: \"2612d124-cc29-46af-bad1-f8af29bee00f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"
Nov 02 17:49:00.287859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.251093    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2612d124-cc29-46af-bad1-f8af29bee00f-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr\" (UID: \"2612d124-cc29-46af-bad1-f8af29bee00f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"
Nov 02 17:49:00.435842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.402969    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"
Nov 02 17:49:00.495164 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.495131    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6c2ls\" (UniqueName: \"kubernetes.io/projected/b3bc4897-c112-464b-8072-62554dcc476b-kube-api-access-6c2ls\") pod \"payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr\" (UID: \"b3bc4897-c112-464b-8072-62554dcc476b\") " pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"
Nov 02 17:49:00.495645 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.495617    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3bc4897-c112-464b-8072-62554dcc476b-plugins-config-volume\") pod \"payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr\" (UID: \"b3bc4897-c112-464b-8072-62554dcc476b\") " pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"
Nov 02 17:49:00.652045 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.651960    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"]
Nov 02 17:49:00.709240 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.709116    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6c2ls\" (UniqueName: \"kubernetes.io/projected/b3bc4897-c112-464b-8072-62554dcc476b-kube-api-access-6c2ls\") pod \"payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr\" (UID: \"b3bc4897-c112-464b-8072-62554dcc476b\") " pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"
Nov 02 17:49:00.709601 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.709577    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3bc4897-c112-464b-8072-62554dcc476b-plugins-config-volume\") pod \"payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr\" (UID: \"b3bc4897-c112-464b-8072-62554dcc476b\") " pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"
Nov 02 17:49:00.710779 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.710754    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3bc4897-c112-464b-8072-62554dcc476b-plugins-config-volume\") pod \"payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr\" (UID: \"b3bc4897-c112-464b-8072-62554dcc476b\") " pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"
Nov 02 17:49:00.744499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.744466    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6c2ls\" (UniqueName: \"kubernetes.io/projected/b3bc4897-c112-464b-8072-62554dcc476b-kube-api-access-6c2ls\") pod \"payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr\" (UID: \"b3bc4897-c112-464b-8072-62554dcc476b\") " pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"
Nov 02 17:49:00.758694 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.758658    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:00.792950 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.768902    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hck44\" (UniqueName: \"kubernetes.io/projected/2612d124-cc29-46af-bad1-f8af29bee00f-kube-api-access-hck44\") pod \"payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr\" (UID: \"2612d124-cc29-46af-bad1-f8af29bee00f\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"
Nov 02 17:49:00.948914 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.948882    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"
Nov 02 17:49:00.993869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.993808    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:49:00.995072 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.955511    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:00.995687 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.957164    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"
Nov 02 17:49:00.995904 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:00.949749    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:49:01.014960 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:01.014914    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:49:01.039517 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:01.039474    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:49:01.446951 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:01.446489    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:49:01.512961 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:01.494958    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 17:49:01.764988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:01.739316    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:49:02.076115 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:02.050613    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:49:02.178851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:02.136126    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 17:49:02.459901 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:02.179185    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:49:02.674021 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:02.179224    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:49:02.724731 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:02.629733    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:49:02.726012 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:02.675122    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:49:02.792558 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:02.792513    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 17:49:02.913424 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:02.892351    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:02.993016 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:49:02.959243    4456 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.345s"
Nov 02 17:49:02.993919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:02.993894    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:49:03.035605 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.035573    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:49:03.052973 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.052926    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:49:03.061973 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.061914    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:49:03.062167 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.062137    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:49:03.090013 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.062314    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:49:03.110110 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.110058    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:49:03.183871 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.161387    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:49:03.259720 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.259636    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"]
Nov 02 17:49:03.266613 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.266582    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"]
Nov 02 17:49:03.325839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.273075    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj" podUID="22b39708-251c-44ad-9b48-2f3c71519131" containerName="istio-proxy" containerID="cri-o://eb26dc382b0920d691696f96b4ff0e70ad2fad6dae676cf773938a373d8aa28d" gracePeriod=30
Nov 02 17:49:03.402841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.340231    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:49:03.414515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.408982    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:49:03.439080 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.438917    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b45b8eb8-wrslt"]
Nov 02 17:49:03.517395 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.517245    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:03.617877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.614464    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:49:03.761804 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.748200    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 17:49:03.815003 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.792226    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:49:03.902223 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.901560    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cbd5984-ab35-4d6f-8aca-b71af0b88ae9" path="/var/lib/kubelet/pods/8cbd5984-ab35-4d6f-8aca-b71af0b88ae9/volumes"
Nov 02 17:49:03.902223 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.902090    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:49:03.902223 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.902157    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:49:03.902223 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.902201    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:49:03.944660 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.944611    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"]
Nov 02 17:49:03.951119 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.951082    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 17:49:03.960383 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.960310    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:49:03.960922 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.960902    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"]
Nov 02 17:49:03.970783 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.970765    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-f59572\""
Nov 02 17:49:03.976062 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.976036    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"]
Nov 02 17:49:03.989931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:03.989066    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"
Nov 02 17:49:04.031675 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.031647    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"]
Nov 02 17:49:04.039829 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.039793    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-f59572-dockercfg-s5jw7\""
Nov 02 17:49:04.045056 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.045033    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"]
Nov 02 17:49:04.074609 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.074534    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"]
Nov 02 17:49:04.128984 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.128117    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a2dbe6-baeb-40c4-be0f-835239be331f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5\" (UID: \"02a2dbe6-baeb-40c4-be0f-835239be331f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"
Nov 02 17:49:04.128984 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.128268    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccrc9\" (UniqueName: \"kubernetes.io/projected/02a2dbe6-baeb-40c4-be0f-835239be331f-kube-api-access-ccrc9\") pod \"payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5\" (UID: \"02a2dbe6-baeb-40c4-be0f-835239be331f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"
Nov 02 17:49:04.128984 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.128320    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc682b81-5f9e-4cd1-98ae-d32e89878ea3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb\" (UID: \"cc682b81-5f9e-4cd1-98ae-d32e89878ea3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 17:49:04.128984 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.128433    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cssvv\" (UniqueName: \"kubernetes.io/projected/cc682b81-5f9e-4cd1-98ae-d32e89878ea3-kube-api-access-cssvv\") pod \"payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb\" (UID: \"cc682b81-5f9e-4cd1-98ae-d32e89878ea3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 17:49:04.239868 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.232918    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccrc9\" (UniqueName: \"kubernetes.io/projected/02a2dbe6-baeb-40c4-be0f-835239be331f-kube-api-access-ccrc9\") pod \"payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5\" (UID: \"02a2dbe6-baeb-40c4-be0f-835239be331f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"
Nov 02 17:49:04.239868 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.232986    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc682b81-5f9e-4cd1-98ae-d32e89878ea3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb\" (UID: \"cc682b81-5f9e-4cd1-98ae-d32e89878ea3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 17:49:04.239868 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.233087    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cssvv\" (UniqueName: \"kubernetes.io/projected/cc682b81-5f9e-4cd1-98ae-d32e89878ea3-kube-api-access-cssvv\") pod \"payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb\" (UID: \"cc682b81-5f9e-4cd1-98ae-d32e89878ea3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 17:49:04.239868 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.233308    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a2dbe6-baeb-40c4-be0f-835239be331f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5\" (UID: \"02a2dbe6-baeb-40c4-be0f-835239be331f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"
Nov 02 17:49:04.240256 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.240223    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc682b81-5f9e-4cd1-98ae-d32e89878ea3-plugins-config-volume\") pod \"payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb\" (UID: \"cc682b81-5f9e-4cd1-98ae-d32e89878ea3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 17:49:04.247020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.246961    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a2dbe6-baeb-40c4-be0f-835239be331f-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5\" (UID: \"02a2dbe6-baeb-40c4-be0f-835239be331f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"
Nov 02 17:49:04.267265 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.267231    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cssvv\" (UniqueName: \"kubernetes.io/projected/cc682b81-5f9e-4cd1-98ae-d32e89878ea3-kube-api-access-cssvv\") pod \"payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb\" (UID: \"cc682b81-5f9e-4cd1-98ae-d32e89878ea3\") " pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 17:49:04.268410 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.268302    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccrc9\" (UniqueName: \"kubernetes.io/projected/02a2dbe6-baeb-40c4-be0f-835239be331f-kube-api-access-ccrc9\") pod \"payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5\" (UID: \"02a2dbe6-baeb-40c4-be0f-835239be331f\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"
Nov 02 17:49:04.276186 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.276160    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 17:49:04.474862 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.474809    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"
Nov 02 17:49:04.530721 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.509726    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:04.737801 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.737757    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"]
Nov 02 17:49:04.835757 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.835630    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" event={"ID":"cc682b81-5f9e-4cd1-98ae-d32e89878ea3","Type":"ContainerStarted","Data":"385fb2039950420031b2c48061b1d32b4c8aa2739ace21371a79a83141090739"}
Nov 02 17:49:04.838981 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.838118    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr" event={"ID":"2612d124-cc29-46af-bad1-f8af29bee00f","Type":"ContainerStarted","Data":"80c4276055304e92ba37840b9e20ba31285ac65a521903b6cb1c0f3c44a44077"}
Nov 02 17:49:04.838981 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.838162    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr" event={"ID":"2612d124-cc29-46af-bad1-f8af29bee00f","Type":"ContainerStarted","Data":"7fb75d74f8fa337c083e869c783a7ae20c15153f051ee26b17392edde937c6ce"}
Nov 02 17:49:04.838981 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.838948    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"
Nov 02 17:49:04.840783 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.840714    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr" event={"ID":"b3bc4897-c112-464b-8072-62554dcc476b","Type":"ContainerStarted","Data":"2cdf2586ed8874a81d7034e27cd512ad9c75703935d263c4a2cf909c69ea362b"}
Nov 02 17:49:04.840783 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.840755    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr" event={"ID":"b3bc4897-c112-464b-8072-62554dcc476b","Type":"ContainerStarted","Data":"0da986c6d24244bf8e79849f034ba5bfb7c2a0f705aca10bcd16b8689c2fdd94"}
Nov 02 17:49:04.841056 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.840795    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"
Nov 02 17:49:04.917630 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.917581    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr" podStartSLOduration=5.917566446 podStartE2EDuration="5.917566446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:04.894412654 +0000 UTC m=+1161.385475807" watchObservedRunningTime="2026-11-02 17:49:04.917566446 +0000 UTC m=+1161.408629647"
Nov 02 17:49:04.947934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:04.947885    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr" podStartSLOduration=5.947871513 podStartE2EDuration="5.947871513s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:48:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:04.94194712 +0000 UTC m=+1161.433010316" watchObservedRunningTime="2026-11-02 17:49:04.947871513 +0000 UTC m=+1161.438934675"
Nov 02 17:49:05.021460 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:05.021427    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"]
Nov 02 17:49:05.022602 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:49:05.022575    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod02a2dbe6_baeb_40c4_be0f_835239be331f.slice/crio-099e22181ceccbf59f14ab74c24153a9fbe4ff720a3ff4ff9e26d14fe45f9fc4 WatchSource:0}: Error finding container 099e22181ceccbf59f14ab74c24153a9fbe4ff720a3ff4ff9e26d14fe45f9fc4: Status 404 returned error can't find the container with id 099e22181ceccbf59f14ab74c24153a9fbe4ff720a3ff4ff9e26d14fe45f9fc4
Nov 02 17:49:05.851182 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:05.851143    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" event={"ID":"cc682b81-5f9e-4cd1-98ae-d32e89878ea3","Type":"ContainerStarted","Data":"865d4366efe4897f0fbc45fe62ca1b7caf82220eebf3378b67df5abe034fc0ed"}
Nov 02 17:49:05.851635 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:05.851203    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 17:49:05.853135 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:05.853106    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5" event={"ID":"02a2dbe6-baeb-40c4-be0f-835239be331f","Type":"ContainerStarted","Data":"ac1aa45ece34b79f424daed8874251c74338193ccb0a28d4a390b8a7bb7763f2"}
Nov 02 17:49:05.853289 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:05.853154    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5" event={"ID":"02a2dbe6-baeb-40c4-be0f-835239be331f","Type":"ContainerStarted","Data":"099e22181ceccbf59f14ab74c24153a9fbe4ff720a3ff4ff9e26d14fe45f9fc4"}
Nov 02 17:49:05.853289 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:05.853230    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"
Nov 02 17:49:05.962722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:05.962652    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" podStartSLOduration=2.962636848 podStartE2EDuration="2.962636848s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:05.916029459 +0000 UTC m=+1162.407092621" watchObservedRunningTime="2026-11-02 17:49:05.962636848 +0000 UTC m=+1162.453700009"
Nov 02 17:49:06.002264 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:06.002185    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5" podStartSLOduration=3.002159844 podStartE2EDuration="3.002159844s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:05.959953994 +0000 UTC m=+1162.451017187" watchObservedRunningTime="2026-11-02 17:49:06.002159844 +0000 UTC m=+1162.493223005"
Nov 02 17:49:06.478006 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:06.477957    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:08.454916 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:08.454318    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:08.895578 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:08.895539    4456 generic.go:356] "Generic (PLEG): container finished" podID="22b39708-251c-44ad-9b48-2f3c71519131" containerID="eb26dc382b0920d691696f96b4ff0e70ad2fad6dae676cf773938a373d8aa28d" exitCode=0
Nov 02 17:49:08.896035 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:08.896008    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj" event={"ID":"22b39708-251c-44ad-9b48-2f3c71519131","Type":"ContainerDied","Data":"eb26dc382b0920d691696f96b4ff0e70ad2fad6dae676cf773938a373d8aa28d"}
Nov 02 17:49:09.034014 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.033980    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:49:09.089997 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.089956    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b39708-251c-44ad-9b48-2f3c71519131-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b39708-251c-44ad-9b48-2f3c71519131-istiod-ca-cert\") pod \"22b39708-251c-44ad-9b48-2f3c71519131\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") "
Nov 02 17:49:09.090183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.090072    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-istio-data\") pod \"22b39708-251c-44ad-9b48-2f3c71519131\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") "
Nov 02 17:49:09.090183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.090131    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b39708-251c-44ad-9b48-2f3c71519131-kube-api-access-4w5gc\" (UniqueName: \"kubernetes.io/projected/22b39708-251c-44ad-9b48-2f3c71519131-kube-api-access-4w5gc\") pod \"22b39708-251c-44ad-9b48-2f3c71519131\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") "
Nov 02 17:49:09.090183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.090180    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-istio-envoy\") pod \"22b39708-251c-44ad-9b48-2f3c71519131\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") "
Nov 02 17:49:09.090379 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.090341    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-workload-socket\") pod \"22b39708-251c-44ad-9b48-2f3c71519131\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") "
Nov 02 17:49:09.090467 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.090403    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-istio-data" pod "22b39708-251c-44ad-9b48-2f3c71519131" (UID: "22b39708-251c-44ad-9b48-2f3c71519131"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:09.090467 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.090433    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-credential-socket\") pod \"22b39708-251c-44ad-9b48-2f3c71519131\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") "
Nov 02 17:49:09.090467 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.090436    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b39708-251c-44ad-9b48-2f3c71519131-istiod-ca-cert" pod "22b39708-251c-44ad-9b48-2f3c71519131" (UID: "22b39708-251c-44ad-9b48-2f3c71519131"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:49:09.090625 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.090466    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-workload-certs\") pod \"22b39708-251c-44ad-9b48-2f3c71519131\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") "
Nov 02 17:49:09.090625 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.090512    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/22b39708-251c-44ad-9b48-2f3c71519131-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b39708-251c-44ad-9b48-2f3c71519131-istio-podinfo\") pod \"22b39708-251c-44ad-9b48-2f3c71519131\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") "
Nov 02 17:49:09.090625 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.090560    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b39708-251c-44ad-9b48-2f3c71519131-istio-token\" (UniqueName: \"kubernetes.io/projected/22b39708-251c-44ad-9b48-2f3c71519131-istio-token\") pod \"22b39708-251c-44ad-9b48-2f3c71519131\" (UID: \"22b39708-251c-44ad-9b48-2f3c71519131\") "
Nov 02 17:49:09.090625 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.090617    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-workload-socket" pod "22b39708-251c-44ad-9b48-2f3c71519131" (UID: "22b39708-251c-44ad-9b48-2f3c71519131"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:09.090845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.090753    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-credential-socket" pod "22b39708-251c-44ad-9b48-2f3c71519131" (UID: "22b39708-251c-44ad-9b48-2f3c71519131"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:09.090903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.090879    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-workload-certs" pod "22b39708-251c-44ad-9b48-2f3c71519131" (UID: "22b39708-251c-44ad-9b48-2f3c71519131"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:09.091537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.091515    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/22b39708-251c-44ad-9b48-2f3c71519131-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:09.091537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.091536    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:09.091682 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.091545    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:09.091682 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.091553    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:09.091682 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.091561    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:09.101394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.101353    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b39708-251c-44ad-9b48-2f3c71519131-istio-token" pod "22b39708-251c-44ad-9b48-2f3c71519131" (UID: "22b39708-251c-44ad-9b48-2f3c71519131"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:09.101655 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.101619    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-istio-envoy" pod "22b39708-251c-44ad-9b48-2f3c71519131" (UID: "22b39708-251c-44ad-9b48-2f3c71519131"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:09.101892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.101869    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b39708-251c-44ad-9b48-2f3c71519131-kube-api-access-4w5gc" pod "22b39708-251c-44ad-9b48-2f3c71519131" (UID: "22b39708-251c-44ad-9b48-2f3c71519131"). InnerVolumeSpecName "kube-api-access-4w5gc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:09.113098 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.113009    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/22b39708-251c-44ad-9b48-2f3c71519131-istio-podinfo" pod "22b39708-251c-44ad-9b48-2f3c71519131" (UID: "22b39708-251c-44ad-9b48-2f3c71519131"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:49:09.193665 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.193629    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/22b39708-251c-44ad-9b48-2f3c71519131-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:09.193665 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.193665    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/22b39708-251c-44ad-9b48-2f3c71519131-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:09.193881 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.193679    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4w5gc\" (UniqueName: \"kubernetes.io/projected/22b39708-251c-44ad-9b48-2f3c71519131-kube-api-access-4w5gc\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:09.193881 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.193693    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/22b39708-251c-44ad-9b48-2f3c71519131-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:09.908849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.908745    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"
Nov 02 17:49:09.908849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.908755    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj" event={"ID":"22b39708-251c-44ad-9b48-2f3c71519131","Type":"ContainerDied","Data":"c3ee98a2f153d0f496edfe95bba4b2fbf19a84c33456c0ccc6eb86e4eac9cc3a"}
Nov 02 17:49:09.909444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.908889    4456 scope.go:122] "RemoveContainer" containerID="eb26dc382b0920d691696f96b4ff0e70ad2fad6dae676cf773938a373d8aa28d"
Nov 02 17:49:09.978071 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.978042    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"]
Nov 02 17:49:09.990738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:09.990710    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-qhzkj"]
Nov 02 17:49:10.037462 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:10.037420    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"]
Nov 02 17:49:10.042202 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:10.042177    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4ef68676-gm2xp"]
Nov 02 17:49:10.192174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:10.186531    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:11.618104 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:11.618072    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1fa2ec57-7deb-424a-ad6b-71dd8caf324c" path="/var/lib/kubelet/pods/1fa2ec57-7deb-424a-ad6b-71dd8caf324c/volumes"
Nov 02 17:49:11.618642 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:11.618627    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b39708-251c-44ad-9b48-2f3c71519131" path="/var/lib/kubelet/pods/22b39708-251c-44ad-9b48-2f3c71519131/volumes"
Nov 02 17:49:11.863278 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:11.863224    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:13.959304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:13.959269    4456 generic.go:356] "Generic (PLEG): container finished" podID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerID="1a22dd302b6925abeb1cc285dfdf5df0473f58e4b460c5dab034c0a14fa18275" exitCode=0
Nov 02 17:49:13.960393 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:13.959347    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" event={"ID":"2dc48441-ccc0-47fc-abe7-7c327559eba4","Type":"ContainerDied","Data":"1a22dd302b6925abeb1cc285dfdf5df0473f58e4b460c5dab034c0a14fa18275"}
Nov 02 17:49:13.960393 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:13.960036    4456 scope.go:122] "RemoveContainer" containerID="1a22dd302b6925abeb1cc285dfdf5df0473f58e4b460c5dab034c0a14fa18275"
Nov 02 17:49:13.962034 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:13.962003    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:13.962132 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:13.962075    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:13.962132 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:13.962107    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:14.290319 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:14.290219    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:14.974496 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:14.974456    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" event={"ID":"2dc48441-ccc0-47fc-abe7-7c327559eba4","Type":"ContainerStarted","Data":"73442968debcfc6a6635d7a8712e48213ab8ebc83bdaf93b458f2f9b8b758f89"}
Nov 02 17:49:15.361666 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:15.361583    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:49:15.362873 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:15.362839    4456 patch_prober.go:29] interesting pod/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 02 17:49:15.363011 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:15.362892    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 02 17:49:15.643902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:15.642427    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:15.856445 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:15.856417    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"
Nov 02 17:49:15.856620 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:15.856482    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"
Nov 02 17:49:16.116841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:16.116713    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:49:16.168139 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:49:16.168096    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:49:16.361142 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:16.361111    4456 patch_prober.go:29] interesting pod/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 02 17:49:16.361318 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:16.361163    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 02 17:49:16.867755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:16.867726    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"
Nov 02 17:49:16.868171 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:16.868150    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 17:49:17.020332 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:17.020295    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"]
Nov 02 17:49:17.020572 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:17.020547    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9" podUID="67472e82-1ea4-420c-81cd-91952f3d0bf7" containerName="istio-proxy" containerID="cri-o://20f6e7886d8d5f395dba728a4d5be467700d43e94ae5e95c113038277f47ab29" gracePeriod=30
Nov 02 17:49:17.026588 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:17.026559    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"]
Nov 02 17:49:17.035757 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:17.035720    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh" podUID="c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" containerName="istio-proxy" containerID="cri-o://4a2e28ca1c905b35f067d73a0dede8d3690199a6c0405b32fb8b66f98931f6af" gracePeriod=30
Nov 02 17:49:17.091875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:17.082068    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"]
Nov 02 17:49:17.091875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:17.082410    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn" podUID="388855a1-d9bc-4803-8230-a981530d412d" containerName="istio-proxy" containerID="cri-o://00866cb105e953f04cf8af424a47330909ec148a18a700abad12199e3f06b775" gracePeriod=30
Nov 02 17:49:17.362011 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:17.361976    4456 patch_prober.go:29] interesting pod/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 02 17:49:17.362436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:17.362032    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 02 17:49:17.396076 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:17.396040    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:18.362155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:18.362109    4456 patch_prober.go:29] interesting pod/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 02 17:49:18.362599 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:18.362174    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 02 17:49:18.905488 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:18.905440    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:19.168770 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:19.168610    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:19.375044 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:19.375009    4456 patch_prober.go:29] interesting pod/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused" start-of-body=
Nov 02 17:49:19.392243 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:19.392206    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": dial tcp 10.134.0.126:15021: connect: connection refused"
Nov 02 17:49:20.366434 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:20.366404    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:49:20.367835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:20.367794    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 17:49:20.663170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:20.663075    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:21.264210 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.264107    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:21.381846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.330257    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"]
Nov 02 17:49:21.381846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.368333    4456 cpu_manager.go:414] "containerMap: removing container" podUID="22b39708-251c-44ad-9b48-2f3c71519131" containerName="istio-proxy"
Nov 02 17:49:21.381846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.368361    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b39708-251c-44ad-9b48-2f3c71519131" containerName="istio-proxy"
Nov 02 17:49:21.381846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.368564    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b39708-251c-44ad-9b48-2f3c71519131" containerName="istio-proxy"
Nov 02 17:49:21.418287 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.418219    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"]
Nov 02 17:49:21.418466 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.418318    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.429009 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.428983    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:21.436997 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.436975    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-f99b8b-openshift-default-dockercfg-wc79k\""
Nov 02 17:49:21.515796 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.515705    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rctlf\" (UniqueName: \"kubernetes.io/projected/98c10ff3-d496-4336-a2cd-921042c719d3-kube-api-access-rctlf\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.515796 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.515766    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c10ff3-d496-4336-a2cd-921042c719d3-istio-podinfo\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.516241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.515911    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-credential-socket\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.516241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.515948    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c10ff3-d496-4336-a2cd-921042c719d3-istiod-ca-cert\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.516241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.515996    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-workload-socket\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.517066 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.516995    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-istio-data\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.517410 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.517126    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-istio-envoy\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.517410 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.517167    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-workload-certs\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.517410 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.517259    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c10ff3-d496-4336-a2cd-921042c719d3-istio-token\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.622774 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.622735    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rctlf\" (UniqueName: \"kubernetes.io/projected/98c10ff3-d496-4336-a2cd-921042c719d3-kube-api-access-rctlf\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.622998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.622791    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c10ff3-d496-4336-a2cd-921042c719d3-istio-podinfo\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.622998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.622900    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-credential-socket\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.622998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.622932    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c10ff3-d496-4336-a2cd-921042c719d3-istiod-ca-cert\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.622998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.622977    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-workload-socket\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.623236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.623009    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-istio-data\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.623236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.623042    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-istio-envoy\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.623236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.623073    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-workload-certs\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.623236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.623159    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c10ff3-d496-4336-a2cd-921042c719d3-istio-token\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.623466 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.623440    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-workload-socket\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.623598 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.623575    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-credential-socket\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.623709 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.623680    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c10ff3-d496-4336-a2cd-921042c719d3-istiod-ca-cert\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.623899 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.623875    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-workload-certs\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.624057 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.623922    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-istio-data\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.629690 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.629659    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c10ff3-d496-4336-a2cd-921042c719d3-istio-podinfo\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.631698 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.630728    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-istio-envoy\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.659121 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.654793    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rctlf\" (UniqueName: \"kubernetes.io/projected/98c10ff3-d496-4336-a2cd-921042c719d3-kube-api-access-rctlf\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.660968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.660938    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c10ff3-d496-4336-a2cd-921042c719d3-istio-token\") pod \"e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") " pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:21.739035 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:21.739000    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:22.111999 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.106961    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:22.127566 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.127527    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"]
Nov 02 17:49:22.167435 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.167392    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:22.167553 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.167490    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:22.167553 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.167532    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:22.374154 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.374124    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:49:22.524520 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.524489    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:49:22.554662 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.554613    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-kube-api-access-dhkgd\" (UniqueName: \"kubernetes.io/projected/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-kube-api-access-dhkgd\") pod \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") "
Nov 02 17:49:22.555026 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.555000    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-podinfo\") pod \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") "
Nov 02 17:49:22.555247 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.555229    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-workload-socket\") pod \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") "
Nov 02 17:49:22.555411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.555395    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-token\" (UniqueName: \"kubernetes.io/projected/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-token\") pod \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") "
Nov 02 17:49:22.555539 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.555523    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istiod-ca-cert\") pod \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") "
Nov 02 17:49:22.555861 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.555776    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-workload-certs\") pod \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") "
Nov 02 17:49:22.556109 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.556088    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-data\") pod \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") "
Nov 02 17:49:22.568217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.566933    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-credential-socket" pod "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" (UID: "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:22.568217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.566997    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-workload-socket" pod "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" (UID: "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:22.568217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.567919    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-data" pod "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" (UID: "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:22.568217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.568094    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istiod-ca-cert" pod "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" (UID: "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:49:22.568858 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.568836    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-workload-certs" pod "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" (UID: "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:22.569260 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.569227    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-kube-api-access-dhkgd" pod "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" (UID: "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b"). InnerVolumeSpecName "kube-api-access-dhkgd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:22.593873 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.592276    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-token" pod "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" (UID: "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:22.611009 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.600572    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-podinfo" pod "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" (UID: "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:49:22.611009 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.609995    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-credential-socket\") pod \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") "
Nov 02 17:49:22.617445 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.615882    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67472e82-1ea4-420c-81cd-91952f3d0bf7-kube-api-access-2mnkj\" (UniqueName: \"kubernetes.io/projected/67472e82-1ea4-420c-81cd-91952f3d0bf7-kube-api-access-2mnkj\") pod \"67472e82-1ea4-420c-81cd-91952f3d0bf7\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") "
Nov 02 17:49:22.617706 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.617686    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-envoy\") pod \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\" (UID: \"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b\") "
Nov 02 17:49:22.618987 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.618963    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.619142 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.619121    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.619251 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.619240    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.619351 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.619329    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.619476 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.619464    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.619574 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.619563    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.619657 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.619645    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dhkgd\" (UniqueName: \"kubernetes.io/projected/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-kube-api-access-dhkgd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.619737 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.619727    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.619826 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.619046    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67472e82-1ea4-420c-81cd-91952f3d0bf7-kube-api-access-2mnkj" pod "67472e82-1ea4-420c-81cd-91952f3d0bf7" (UID: "67472e82-1ea4-420c-81cd-91952f3d0bf7"). InnerVolumeSpecName "kube-api-access-2mnkj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:22.622353 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.622329    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-envoy" pod "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" (UID: "c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:22.702691 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.702656    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.720734    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-token\" (UniqueName: \"kubernetes.io/projected/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-token\") pod \"67472e82-1ea4-420c-81cd-91952f3d0bf7\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") "
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.720973    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-envoy\") pod \"67472e82-1ea4-420c-81cd-91952f3d0bf7\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") "
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.721130    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-data\") pod \"67472e82-1ea4-420c-81cd-91952f3d0bf7\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") "
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.721234    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67472e82-1ea4-420c-81cd-91952f3d0bf7-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67472e82-1ea4-420c-81cd-91952f3d0bf7-istiod-ca-cert\") pod \"67472e82-1ea4-420c-81cd-91952f3d0bf7\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") "
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.721269    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-credential-socket\") pod \"67472e82-1ea4-420c-81cd-91952f3d0bf7\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") "
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.721355    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-workload-certs\") pod \"67472e82-1ea4-420c-81cd-91952f3d0bf7\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") "
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.721504    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-workload-socket\") pod \"67472e82-1ea4-420c-81cd-91952f3d0bf7\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") "
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.721605    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-podinfo\") pod \"67472e82-1ea4-420c-81cd-91952f3d0bf7\" (UID: \"67472e82-1ea4-420c-81cd-91952f3d0bf7\") "
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.722621    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-data" pod "67472e82-1ea4-420c-81cd-91952f3d0bf7" (UID: "67472e82-1ea4-420c-81cd-91952f3d0bf7"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.722679    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mnkj\" (UniqueName: \"kubernetes.io/projected/67472e82-1ea4-420c-81cd-91952f3d0bf7-kube-api-access-2mnkj\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.722704    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.722802    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-workload-certs" pod "67472e82-1ea4-420c-81cd-91952f3d0bf7" (UID: "67472e82-1ea4-420c-81cd-91952f3d0bf7"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.722997    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-credential-socket" pod "67472e82-1ea4-420c-81cd-91952f3d0bf7" (UID: "67472e82-1ea4-420c-81cd-91952f3d0bf7"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.724075    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-workload-socket" pod "67472e82-1ea4-420c-81cd-91952f3d0bf7" (UID: "67472e82-1ea4-420c-81cd-91952f3d0bf7"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.724873    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67472e82-1ea4-420c-81cd-91952f3d0bf7-istiod-ca-cert" pod "67472e82-1ea4-420c-81cd-91952f3d0bf7" (UID: "67472e82-1ea4-420c-81cd-91952f3d0bf7"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.727085    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-token" pod "67472e82-1ea4-420c-81cd-91952f3d0bf7" (UID: "67472e82-1ea4-420c-81cd-91952f3d0bf7"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:22.731841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.730687    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-podinfo" pod "67472e82-1ea4-420c-81cd-91952f3d0bf7" (UID: "67472e82-1ea4-420c-81cd-91952f3d0bf7"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:49:22.735457 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.732488    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-envoy" pod "67472e82-1ea4-420c-81cd-91952f3d0bf7" (UID: "67472e82-1ea4-420c-81cd-91952f3d0bf7"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:22.823791 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.823703    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/388855a1-d9bc-4803-8230-a981530d412d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388855a1-d9bc-4803-8230-a981530d412d-istio-podinfo\") pod \"388855a1-d9bc-4803-8230-a981530d412d\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") "
Nov 02 17:49:22.825206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.824334    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-credential-socket\") pod \"388855a1-d9bc-4803-8230-a981530d412d\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") "
Nov 02 17:49:22.825206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.824531    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-workload-socket\") pod \"388855a1-d9bc-4803-8230-a981530d412d\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") "
Nov 02 17:49:22.825206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.824569    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388855a1-d9bc-4803-8230-a981530d412d-istio-token\" (UniqueName: \"kubernetes.io/projected/388855a1-d9bc-4803-8230-a981530d412d-istio-token\") pod \"388855a1-d9bc-4803-8230-a981530d412d\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") "
Nov 02 17:49:22.825206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.824712    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-istio-envoy\") pod \"388855a1-d9bc-4803-8230-a981530d412d\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") "
Nov 02 17:49:22.825206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.824754    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-istio-data\") pod \"388855a1-d9bc-4803-8230-a981530d412d\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") "
Nov 02 17:49:22.825206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.824781    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/388855a1-d9bc-4803-8230-a981530d412d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388855a1-d9bc-4803-8230-a981530d412d-istiod-ca-cert\") pod \"388855a1-d9bc-4803-8230-a981530d412d\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") "
Nov 02 17:49:22.825206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.824906    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/388855a1-d9bc-4803-8230-a981530d412d-kube-api-access-jgj72\" (UniqueName: \"kubernetes.io/projected/388855a1-d9bc-4803-8230-a981530d412d-kube-api-access-jgj72\") pod \"388855a1-d9bc-4803-8230-a981530d412d\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") "
Nov 02 17:49:22.825646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.825244    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-credential-socket" pod "388855a1-d9bc-4803-8230-a981530d412d" (UID: "388855a1-d9bc-4803-8230-a981530d412d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:22.825646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.825257    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-workload-certs\") pod \"388855a1-d9bc-4803-8230-a981530d412d\" (UID: \"388855a1-d9bc-4803-8230-a981530d412d\") "
Nov 02 17:49:22.825646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.825527    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-workload-certs" pod "388855a1-d9bc-4803-8230-a981530d412d" (UID: "388855a1-d9bc-4803-8230-a981530d412d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:22.826247 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.826018    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/388855a1-d9bc-4803-8230-a981530d412d-istiod-ca-cert" pod "388855a1-d9bc-4803-8230-a981530d412d" (UID: "388855a1-d9bc-4803-8230-a981530d412d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:49:22.826443 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.826409    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-istio-data" pod "388855a1-d9bc-4803-8230-a981530d412d" (UID: "388855a1-d9bc-4803-8230-a981530d412d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:22.826733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.826565    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.826733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.826592    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.826733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.826606    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.826733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.826622    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.826733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.826635    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.826733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.826648    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/388855a1-d9bc-4803-8230-a981530d412d-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.826733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.826660    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67472e82-1ea4-420c-81cd-91952f3d0bf7-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.826733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.826673    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.826733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.826686    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.826733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.826699    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.826733 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.826714    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67472e82-1ea4-420c-81cd-91952f3d0bf7-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.827763 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.827709    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-workload-socket" pod "388855a1-d9bc-4803-8230-a981530d412d" (UID: "388855a1-d9bc-4803-8230-a981530d412d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:22.830576 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.830543    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/388855a1-d9bc-4803-8230-a981530d412d-istio-podinfo" pod "388855a1-d9bc-4803-8230-a981530d412d" (UID: "388855a1-d9bc-4803-8230-a981530d412d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:49:22.831238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.831201    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-istio-envoy" pod "388855a1-d9bc-4803-8230-a981530d412d" (UID: "388855a1-d9bc-4803-8230-a981530d412d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:49:22.831547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.831513    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388855a1-d9bc-4803-8230-a981530d412d-istio-token" pod "388855a1-d9bc-4803-8230-a981530d412d" (UID: "388855a1-d9bc-4803-8230-a981530d412d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:22.833630 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.833592    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/388855a1-d9bc-4803-8230-a981530d412d-kube-api-access-jgj72" pod "388855a1-d9bc-4803-8230-a981530d412d" (UID: "388855a1-d9bc-4803-8230-a981530d412d"). InnerVolumeSpecName "kube-api-access-jgj72". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:49:22.929085 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.929049    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.929085 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.929085    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.929291 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.929100    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgj72\" (UniqueName: \"kubernetes.io/projected/388855a1-d9bc-4803-8230-a981530d412d-kube-api-access-jgj72\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.929291 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.929114    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/388855a1-d9bc-4803-8230-a981530d412d-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.929291 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.929129    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/388855a1-d9bc-4803-8230-a981530d412d-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:22.929291 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:22.929143    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/388855a1-d9bc-4803-8230-a981530d412d-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:49:23.041194 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.041152    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"]
Nov 02 17:49:23.045855 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.045827    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e676391c-khfrt"]
Nov 02 17:49:23.110790 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.107865    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:23.182623 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.182581    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7" event={"ID":"98c10ff3-d496-4336-a2cd-921042c719d3","Type":"ContainerStarted","Data":"4c2b669d919f440b588554d96ba166ca71127cc37440ec0db956fe5315cc5fdc"}
Nov 02 17:49:23.182803 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.182657    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7" event={"ID":"98c10ff3-d496-4336-a2cd-921042c719d3","Type":"ContainerStarted","Data":"89e536a98227e233f0423debcecaf77140ae2c4f77dd6711832ca28d5bfba7b5"}
Nov 02 17:49:23.185361 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.185332    4456 generic.go:356] "Generic (PLEG): container finished" podID="388855a1-d9bc-4803-8230-a981530d412d" containerID="00866cb105e953f04cf8af424a47330909ec148a18a700abad12199e3f06b775" exitCode=0
Nov 02 17:49:23.189603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.185534    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"
Nov 02 17:49:23.189603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.185998    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn" event={"ID":"388855a1-d9bc-4803-8230-a981530d412d","Type":"ContainerDied","Data":"00866cb105e953f04cf8af424a47330909ec148a18a700abad12199e3f06b775"}
Nov 02 17:49:23.189603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.187255    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn" event={"ID":"388855a1-d9bc-4803-8230-a981530d412d","Type":"ContainerDied","Data":"533c3a02fda013e09d6b65662f40e33cef6351e093d783c0a32560279ca1d640"}
Nov 02 17:49:23.189603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.187299    4456 scope.go:122] "RemoveContainer" containerID="00866cb105e953f04cf8af424a47330909ec148a18a700abad12199e3f06b775"
Nov 02 17:49:23.197860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.196921    4456 generic.go:356] "Generic (PLEG): container finished" podID="c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" containerID="4a2e28ca1c905b35f067d73a0dede8d3690199a6c0405b32fb8b66f98931f6af" exitCode=0
Nov 02 17:49:23.197860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.197086    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh" event={"ID":"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b","Type":"ContainerDied","Data":"4a2e28ca1c905b35f067d73a0dede8d3690199a6c0405b32fb8b66f98931f6af"}
Nov 02 17:49:23.197860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.197124    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh" event={"ID":"c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b","Type":"ContainerDied","Data":"d5f95f26ddc6f09846b5a6d0e36d62895ad9306c2317168c3620bf370df8baa9"}
Nov 02 17:49:23.197860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.197199    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"
Nov 02 17:49:23.205997 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.205970    4456 generic.go:356] "Generic (PLEG): container finished" podID="67472e82-1ea4-420c-81cd-91952f3d0bf7" containerID="20f6e7886d8d5f395dba728a4d5be467700d43e94ae5e95c113038277f47ab29" exitCode=0
Nov 02 17:49:23.206120 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.206075    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9" event={"ID":"67472e82-1ea4-420c-81cd-91952f3d0bf7","Type":"ContainerDied","Data":"20f6e7886d8d5f395dba728a4d5be467700d43e94ae5e95c113038277f47ab29"}
Nov 02 17:49:23.206120 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.206116    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9" event={"ID":"67472e82-1ea4-420c-81cd-91952f3d0bf7","Type":"ContainerDied","Data":"496be73610e3e6410b4d5484d32c14b99222791e86e8baed864b11bbc7a4c9c3"}
Nov 02 17:49:23.206233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.206130    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"
Nov 02 17:49:23.213251 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.213219    4456 scope.go:122] "RemoveContainer" containerID="00866cb105e953f04cf8af424a47330909ec148a18a700abad12199e3f06b775"
Nov 02 17:49:23.213547 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:49:23.213527    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"00866cb105e953f04cf8af424a47330909ec148a18a700abad12199e3f06b775\": container with ID starting with 00866cb105e953f04cf8af424a47330909ec148a18a700abad12199e3f06b775 not found: ID does not exist" containerID="00866cb105e953f04cf8af424a47330909ec148a18a700abad12199e3f06b775"
Nov 02 17:49:23.213630 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.213558    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"00866cb105e953f04cf8af424a47330909ec148a18a700abad12199e3f06b775"} err="failed to get container status \"00866cb105e953f04cf8af424a47330909ec148a18a700abad12199e3f06b775\": rpc error: code = NotFound desc = could not find container \"00866cb105e953f04cf8af424a47330909ec148a18a700abad12199e3f06b775\": container with ID starting with 00866cb105e953f04cf8af424a47330909ec148a18a700abad12199e3f06b775 not found: ID does not exist"
Nov 02 17:49:23.213630 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.213584    4456 scope.go:122] "RemoveContainer" containerID="4a2e28ca1c905b35f067d73a0dede8d3690199a6c0405b32fb8b66f98931f6af"
Nov 02 17:49:23.226427 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.226406    4456 scope.go:122] "RemoveContainer" containerID="4a2e28ca1c905b35f067d73a0dede8d3690199a6c0405b32fb8b66f98931f6af"
Nov 02 17:49:23.226722 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:49:23.226701    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a2e28ca1c905b35f067d73a0dede8d3690199a6c0405b32fb8b66f98931f6af\": container with ID starting with 4a2e28ca1c905b35f067d73a0dede8d3690199a6c0405b32fb8b66f98931f6af not found: ID does not exist" containerID="4a2e28ca1c905b35f067d73a0dede8d3690199a6c0405b32fb8b66f98931f6af"
Nov 02 17:49:23.226784 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.226729    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a2e28ca1c905b35f067d73a0dede8d3690199a6c0405b32fb8b66f98931f6af"} err="failed to get container status \"4a2e28ca1c905b35f067d73a0dede8d3690199a6c0405b32fb8b66f98931f6af\": rpc error: code = NotFound desc = could not find container \"4a2e28ca1c905b35f067d73a0dede8d3690199a6c0405b32fb8b66f98931f6af\": container with ID starting with 4a2e28ca1c905b35f067d73a0dede8d3690199a6c0405b32fb8b66f98931f6af not found: ID does not exist"
Nov 02 17:49:23.226784 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.226747    4456 scope.go:122] "RemoveContainer" containerID="20f6e7886d8d5f395dba728a4d5be467700d43e94ae5e95c113038277f47ab29"
Nov 02 17:49:23.239687 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.239665    4456 scope.go:122] "RemoveContainer" containerID="20f6e7886d8d5f395dba728a4d5be467700d43e94ae5e95c113038277f47ab29"
Nov 02 17:49:23.240120 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:49:23.240088    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20f6e7886d8d5f395dba728a4d5be467700d43e94ae5e95c113038277f47ab29\": container with ID starting with 20f6e7886d8d5f395dba728a4d5be467700d43e94ae5e95c113038277f47ab29 not found: ID does not exist" containerID="20f6e7886d8d5f395dba728a4d5be467700d43e94ae5e95c113038277f47ab29"
Nov 02 17:49:23.240214 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.240130    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20f6e7886d8d5f395dba728a4d5be467700d43e94ae5e95c113038277f47ab29"} err="failed to get container status \"20f6e7886d8d5f395dba728a4d5be467700d43e94ae5e95c113038277f47ab29\": rpc error: code = NotFound desc = could not find container \"20f6e7886d8d5f395dba728a4d5be467700d43e94ae5e95c113038277f47ab29\": container with ID starting with 20f6e7886d8d5f395dba728a4d5be467700d43e94ae5e95c113038277f47ab29 not found: ID does not exist"
Nov 02 17:49:23.274493 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.274427    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7" podStartSLOduration=2.27440906 podStartE2EDuration="2.27440906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:23.235884877 +0000 UTC m=+1179.726948064" watchObservedRunningTime="2026-11-02 17:49:23.27440906 +0000 UTC m=+1179.765472220"
Nov 02 17:49:23.279159 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.279131    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"]
Nov 02 17:49:23.284262 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.284225    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-vx8fh"]
Nov 02 17:49:23.319314 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.319283    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"]
Nov 02 17:49:23.325003 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.324979    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-2vnmn"]
Nov 02 17:49:23.364991 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.364832    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:23.371025 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.370998    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"]
Nov 02 17:49:23.376998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.376969    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-8hwz9"]
Nov 02 17:49:23.619546 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.619472    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="388855a1-d9bc-4803-8230-a981530d412d" path="/var/lib/kubelet/pods/388855a1-d9bc-4803-8230-a981530d412d/volumes"
Nov 02 17:49:23.619963 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.619949    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="42081d03-198e-437e-893f-ba984a5b5159" path="/var/lib/kubelet/pods/42081d03-198e-437e-893f-ba984a5b5159/volumes"
Nov 02 17:49:23.620393 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.620381    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67472e82-1ea4-420c-81cd-91952f3d0bf7" path="/var/lib/kubelet/pods/67472e82-1ea4-420c-81cd-91952f3d0bf7/volumes"
Nov 02 17:49:23.621172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.621158    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" path="/var/lib/kubelet/pods/c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b/volumes"
Nov 02 17:49:23.739905 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.739874    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:23.746053 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:23.746025    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:24.057951 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:24.057920    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"]
Nov 02 17:49:24.062533 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:24.062508    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5ef19a7c-rtzwp"]
Nov 02 17:49:24.222236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:24.222151    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:24.223484 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:24.223461    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:49:25.101122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:25.101080    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:25.618377 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:25.618340    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a83be4c-c2f5-4392-9a73-ad30b375ae59" path="/var/lib/kubelet/pods/2a83be4c-c2f5-4392-9a73-ad30b375ae59/volumes"
Nov 02 17:49:25.824318 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:25.824288    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:26.072876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:26.071331    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"]
Nov 02 17:49:26.086245 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:26.086206    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f2176fc8-7fkb8"]
Nov 02 17:49:26.269928 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:26.269898    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:26.475301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:26.475263    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:27.119553 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.119520    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"]
Nov 02 17:49:27.120359 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.120336    4456 cpu_manager.go:414] "containerMap: removing container" podUID="c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" containerName="istio-proxy"
Nov 02 17:49:27.120359 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.120359    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" containerName="istio-proxy"
Nov 02 17:49:27.120538 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.120382    4456 cpu_manager.go:414] "containerMap: removing container" podUID="67472e82-1ea4-420c-81cd-91952f3d0bf7" containerName="istio-proxy"
Nov 02 17:49:27.120538 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.120390    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67472e82-1ea4-420c-81cd-91952f3d0bf7" containerName="istio-proxy"
Nov 02 17:49:27.120538 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.120426    4456 cpu_manager.go:414] "containerMap: removing container" podUID="388855a1-d9bc-4803-8230-a981530d412d" containerName="istio-proxy"
Nov 02 17:49:27.120538 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.120433    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="388855a1-d9bc-4803-8230-a981530d412d" containerName="istio-proxy"
Nov 02 17:49:27.120732 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.120659    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="388855a1-d9bc-4803-8230-a981530d412d" containerName="istio-proxy"
Nov 02 17:49:27.120732 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.120688    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5dc9460-ceb7-4c3d-83bc-f038aeb26b8b" containerName="istio-proxy"
Nov 02 17:49:27.120732 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.120699    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="67472e82-1ea4-420c-81cd-91952f3d0bf7" containerName="istio-proxy"
Nov 02 17:49:27.125346 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.125325    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:27.145975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.145938    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"]
Nov 02 17:49:27.151545 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.151503    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-mt-0d1f5d84-serving-cert\""
Nov 02 17:49:27.228463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.228431    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc728ac-8cd4-44e7-b831-8bbc21a40db0-metrics-certs\") pod \"maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:27.228701 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.228678    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g8mkb\" (UniqueName: \"kubernetes.io/projected/edc728ac-8cd4-44e7-b831-8bbc21a40db0-kube-api-access-g8mkb\") pod \"maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:27.228751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.228723    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc728ac-8cd4-44e7-b831-8bbc21a40db0-maas-api-tls\") pod \"maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:27.228858 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.228832    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc728ac-8cd4-44e7-b831-8bbc21a40db0-openshift-service-ca\") pod \"maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:27.298776 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.298743    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:27.330907 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.330872    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc728ac-8cd4-44e7-b831-8bbc21a40db0-metrics-certs\") pod \"maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:27.331093 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.331041    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g8mkb\" (UniqueName: \"kubernetes.io/projected/edc728ac-8cd4-44e7-b831-8bbc21a40db0-kube-api-access-g8mkb\") pod \"maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:27.331093 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.331070    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc728ac-8cd4-44e7-b831-8bbc21a40db0-maas-api-tls\") pod \"maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:27.331223 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.331100    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc728ac-8cd4-44e7-b831-8bbc21a40db0-openshift-service-ca\") pod \"maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:27.331852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.331767    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc728ac-8cd4-44e7-b831-8bbc21a40db0-openshift-service-ca\") pod \"maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:27.333569 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.333549    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc728ac-8cd4-44e7-b831-8bbc21a40db0-metrics-certs\") pod \"maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:27.333655 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.333611    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc728ac-8cd4-44e7-b831-8bbc21a40db0-maas-api-tls\") pod \"maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:27.363875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.356296    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g8mkb\" (UniqueName: \"kubernetes.io/projected/edc728ac-8cd4-44e7-b831-8bbc21a40db0-kube-api-access-g8mkb\") pod \"maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") " pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:27.441666 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.441628    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:27.479677 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.479543    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:27.631876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.631836    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d73ea517-8865-490e-a7bd-24d164f8c1a1" path="/var/lib/kubelet/pods/d73ea517-8865-490e-a7bd-24d164f8c1a1/volumes"
Nov 02 17:49:27.779067 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:49:27.779026    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podedc728ac_8cd4_44e7_b831_8bbc21a40db0.slice/crio-9ffe0e4ce8b8f0f099a5f2067a0d8208b4efbc1758e849d41e437213a80c80de WatchSource:0}: Error finding container 9ffe0e4ce8b8f0f099a5f2067a0d8208b4efbc1758e849d41e437213a80c80de: Status 404 returned error can't find the container with id 9ffe0e4ce8b8f0f099a5f2067a0d8208b4efbc1758e849d41e437213a80c80de
Nov 02 17:49:27.780037 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:27.779925    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"]
Nov 02 17:49:28.007353 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:28.007313    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:28.264723 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:28.264690    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p" event={"ID":"edc728ac-8cd4-44e7-b831-8bbc21a40db0","Type":"ContainerStarted","Data":"4bcac2cac206fef1426452fcac125a7e35c994f98f1470b119a94ed2de318f0c"}
Nov 02 17:49:28.264932 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:28.264746    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p" event={"ID":"edc728ac-8cd4-44e7-b831-8bbc21a40db0","Type":"ContainerStarted","Data":"9ffe0e4ce8b8f0f099a5f2067a0d8208b4efbc1758e849d41e437213a80c80de"}
Nov 02 17:49:28.264932 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:28.264805    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:28.312242 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:28.312190    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p" podStartSLOduration=0.953390608 podStartE2EDuration="1.312169949s" totalImagesPullingTime="358.779341ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:27 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:28.309968931 +0000 UTC m=+1184.801032115" watchObservedRunningTime="2026-11-02 17:49:28.312169949 +0000 UTC m=+1184.803233110"
Nov 02 17:49:29.162997 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:29.162872    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:30.447267 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.447229    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:30.562794 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.562761    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"]
Nov 02 17:49:30.567509 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.567487    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"
Nov 02 17:49:30.589229 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.589199    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"]
Nov 02 17:49:30.591134 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.591107    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-f99b8b\""
Nov 02 17:49:30.591502 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.591483    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-f99b8b-dockercfg-nq2lc\""
Nov 02 17:49:30.596325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.596301    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:30.716894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.716720    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee1be5c9-f0b9-48b9-a396-540b41d57247-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f\" (UID: \"ee1be5c9-f0b9-48b9-a396-540b41d57247\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"
Nov 02 17:49:30.716894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.716803    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g2rpd\" (UniqueName: \"kubernetes.io/projected/ee1be5c9-f0b9-48b9-a396-540b41d57247-kube-api-access-g2rpd\") pod \"payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f\" (UID: \"ee1be5c9-f0b9-48b9-a396-540b41d57247\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"
Nov 02 17:49:30.785839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.784115    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"]
Nov 02 17:49:30.794835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.793057    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"
Nov 02 17:49:30.813497 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.803246    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"]
Nov 02 17:49:30.843838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.834721    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee1be5c9-f0b9-48b9-a396-540b41d57247-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f\" (UID: \"ee1be5c9-f0b9-48b9-a396-540b41d57247\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"
Nov 02 17:49:30.843838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.834789    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g2rpd\" (UniqueName: \"kubernetes.io/projected/ee1be5c9-f0b9-48b9-a396-540b41d57247-kube-api-access-g2rpd\") pod \"payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f\" (UID: \"ee1be5c9-f0b9-48b9-a396-540b41d57247\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"
Nov 02 17:49:30.843838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.835947    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee1be5c9-f0b9-48b9-a396-540b41d57247-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f\" (UID: \"ee1be5c9-f0b9-48b9-a396-540b41d57247\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"
Nov 02 17:49:30.944418 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.944378    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24df51d-d414-475e-8850-99604ace85dc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d\" (UID: \"c24df51d-d414-475e-8850-99604ace85dc\") " pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"
Nov 02 17:49:30.944679 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.944656    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hq274\" (UniqueName: \"kubernetes.io/projected/c24df51d-d414-475e-8850-99604ace85dc-kube-api-access-hq274\") pod \"payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d\" (UID: \"c24df51d-d414-475e-8850-99604ace85dc\") " pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"
Nov 02 17:49:30.961746 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:30.955176    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g2rpd\" (UniqueName: \"kubernetes.io/projected/ee1be5c9-f0b9-48b9-a396-540b41d57247-kube-api-access-g2rpd\") pod \"payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f\" (UID: \"ee1be5c9-f0b9-48b9-a396-540b41d57247\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"
Nov 02 17:49:31.047899 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:31.046605    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24df51d-d414-475e-8850-99604ace85dc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d\" (UID: \"c24df51d-d414-475e-8850-99604ace85dc\") " pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"
Nov 02 17:49:31.047899 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:31.046651    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hq274\" (UniqueName: \"kubernetes.io/projected/c24df51d-d414-475e-8850-99604ace85dc-kube-api-access-hq274\") pod \"payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d\" (UID: \"c24df51d-d414-475e-8850-99604ace85dc\") " pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"
Nov 02 17:49:31.048123 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:31.047922    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24df51d-d414-475e-8850-99604ace85dc-plugins-config-volume\") pod \"payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d\" (UID: \"c24df51d-d414-475e-8850-99604ace85dc\") " pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"
Nov 02 17:49:31.062993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:31.062549    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:31.071868 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:31.071693    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hq274\" (UniqueName: \"kubernetes.io/projected/c24df51d-d414-475e-8850-99604ace85dc-kube-api-access-hq274\") pod \"payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d\" (UID: \"c24df51d-d414-475e-8850-99604ace85dc\") " pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"
Nov 02 17:49:31.199924 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:31.199883    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"
Nov 02 17:49:31.221456 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:31.221411    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"
Nov 02 17:49:31.226530 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:31.226493    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:31.591897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:31.591837    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:31.613728 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:49:31.613687    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podee1be5c9_f0b9_48b9_a396_540b41d57247.slice/crio-7ab94e88683e7f62bd9e3d4d02e5c6aaaa89c046e821d9d2fb32fc290161335e WatchSource:0}: Error finding container 7ab94e88683e7f62bd9e3d4d02e5c6aaaa89c046e821d9d2fb32fc290161335e: Status 404 returned error can't find the container with id 7ab94e88683e7f62bd9e3d4d02e5c6aaaa89c046e821d9d2fb32fc290161335e
Nov 02 17:49:31.619142 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:31.619109    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"]
Nov 02 17:49:31.655420 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:49:31.655390    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc24df51d_d414_475e_8850_99604ace85dc.slice/crio-3f95519553b191e2ad95fa866036cbc5594537f5ecce188acc5f598d7cb49a34 WatchSource:0}: Error finding container 3f95519553b191e2ad95fa866036cbc5594537f5ecce188acc5f598d7cb49a34: Status 404 returned error can't find the container with id 3f95519553b191e2ad95fa866036cbc5594537f5ecce188acc5f598d7cb49a34
Nov 02 17:49:31.655614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:31.655587    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"]
Nov 02 17:49:32.049434 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:32.049396    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"]
Nov 02 17:49:32.060444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:32.060411    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-da9459d3-84fdr"]
Nov 02 17:49:32.358376 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:32.358259    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d" event={"ID":"c24df51d-d414-475e-8850-99604ace85dc","Type":"ContainerStarted","Data":"7fe40d10ca68b3279d6e983abecbe433687aaf4f3934af746a67014a9263daa6"}
Nov 02 17:49:32.358376 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:32.358346    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d" event={"ID":"c24df51d-d414-475e-8850-99604ace85dc","Type":"ContainerStarted","Data":"3f95519553b191e2ad95fa866036cbc5594537f5ecce188acc5f598d7cb49a34"}
Nov 02 17:49:32.359687 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:32.359661    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"
Nov 02 17:49:32.363310 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:32.363119    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f" event={"ID":"ee1be5c9-f0b9-48b9-a396-540b41d57247","Type":"ContainerStarted","Data":"2a33e195aadc2cd8b8c7ef4662a0674fbd2ccc0cfc3095d4c12599f1e17bc12c"}
Nov 02 17:49:32.363310 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:32.363171    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f" event={"ID":"ee1be5c9-f0b9-48b9-a396-540b41d57247","Type":"ContainerStarted","Data":"7ab94e88683e7f62bd9e3d4d02e5c6aaaa89c046e821d9d2fb32fc290161335e"}
Nov 02 17:49:32.363310 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:32.363198    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"
Nov 02 17:49:32.407016 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:32.406961    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d" podStartSLOduration=2.406946529 podStartE2EDuration="2.406946529s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:32.401940559 +0000 UTC m=+1188.893003738" watchObservedRunningTime="2026-11-02 17:49:32.406946529 +0000 UTC m=+1188.898009690"
Nov 02 17:49:32.460463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:32.460408    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f" podStartSLOduration=2.460394337 podStartE2EDuration="2.460394337s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:30 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:32.454036651 +0000 UTC m=+1188.945099814" watchObservedRunningTime="2026-11-02 17:49:32.460394337 +0000 UTC m=+1188.951457527"
Nov 02 17:49:33.057938 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:33.057892    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:33.621033 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:33.620987    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="da476c88-7665-46d9-917e-db0100e78636" path="/var/lib/kubelet/pods/da476c88-7665-46d9-917e-db0100e78636/volumes"
Nov 02 17:49:34.279144 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:34.279115    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:49:34.665856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:34.665759    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:34.872131 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:34.872094    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:36.753426 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:36.753391    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:38.668029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:38.667984    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:40.637421 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:40.637304    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:41.515101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:41.515059    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:41.936183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:41.936150    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"]
Nov 02 17:49:41.942537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:41.942515    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:41.958892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:41.958862    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-786c42-openshift-default-dockercfg-b6zlr\""
Nov 02 17:49:41.960784 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:41.960762    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-credential-socket\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:41.961036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:41.961010    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33530571-d61c-488e-be5b-54e298cfd4e4-istio-token\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:41.961190 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:41.961056    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-istio-data\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:41.961316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:41.961296    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-workload-certs\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:41.961472 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:41.961428    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4lt76\" (UniqueName: \"kubernetes.io/projected/33530571-d61c-488e-be5b-54e298cfd4e4-kube-api-access-4lt76\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:41.961768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:41.961747    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33530571-d61c-488e-be5b-54e298cfd4e4-istiod-ca-cert\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:41.961961 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:41.961944    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33530571-d61c-488e-be5b-54e298cfd4e4-istio-podinfo\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:41.962759 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:41.962726    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-workload-socket\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:41.962929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:41.962895    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-istio-envoy\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:41.978516 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:41.978486    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"]
Nov 02 17:49:42.089877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.070550    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-workload-certs\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.089877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.070666    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4lt76\" (UniqueName: \"kubernetes.io/projected/33530571-d61c-488e-be5b-54e298cfd4e4-kube-api-access-4lt76\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.089877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.070758    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33530571-d61c-488e-be5b-54e298cfd4e4-istiod-ca-cert\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.089877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.071204    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33530571-d61c-488e-be5b-54e298cfd4e4-istio-podinfo\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.089877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.071322    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-workload-socket\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.089877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.071459    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-istio-envoy\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.089877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.071558    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-credential-socket\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.089877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.071586    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33530571-d61c-488e-be5b-54e298cfd4e4-istio-token\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.089877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.071619    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-istio-data\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.089877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.072247    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-istio-data\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.089877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.072597    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-workload-certs\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.089877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.073562    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33530571-d61c-488e-be5b-54e298cfd4e4-istiod-ca-cert\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.089877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.076848    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:42.130396 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.119261    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"]
Nov 02 17:49:42.130396 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.119637    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-workload-socket\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.130396 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.130165    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-credential-socket\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.187899 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.187743    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33530571-d61c-488e-be5b-54e298cfd4e4-istio-token\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.281406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.281366    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:42.330841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.325575    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4lt76\" (UniqueName: \"kubernetes.io/projected/33530571-d61c-488e-be5b-54e298cfd4e4-kube-api-access-4lt76\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.450482 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.450391    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-869d79c6-28cxr"]
Nov 02 17:49:42.459768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.457694    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33530571-d61c-488e-be5b-54e298cfd4e4-istio-podinfo\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.459768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.459423    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-istio-envoy\") pod \"e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") " pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.568766 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.568709    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:42.668293 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:42.668260    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:43.058889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:43.058862    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"]
Nov 02 17:49:43.059234 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:49:43.059005    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod33530571_d61c_488e_be5b_54e298cfd4e4.slice/crio-b6c7f4131610b4c354ecc75c2432b628f6982c0362375095c0008a5d465e3d05 WatchSource:0}: Error finding container b6c7f4131610b4c354ecc75c2432b628f6982c0362375095c0008a5d465e3d05: Status 404 returned error can't find the container with id b6c7f4131610b4c354ecc75c2432b628f6982c0362375095c0008a5d465e3d05
Nov 02 17:49:43.061803 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:43.061772    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:43.061909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:43.061888    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:43.061949 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:43.061921    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:43.376621 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:43.376546    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"
Nov 02 17:49:43.561094 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:43.561056    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd" event={"ID":"33530571-d61c-488e-be5b-54e298cfd4e4","Type":"ContainerStarted","Data":"64e1c83b77cad53fbb7765a03a1aae028227679f40fd0cd1509bef2eb59cd3d5"}
Nov 02 17:49:43.561291 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:43.561120    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd" event={"ID":"33530571-d61c-488e-be5b-54e298cfd4e4","Type":"ContainerStarted","Data":"b6c7f4131610b4c354ecc75c2432b628f6982c0362375095c0008a5d465e3d05"}
Nov 02 17:49:43.569989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:43.569967    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:43.642112 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:43.642013    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="372e3319-752f-43bb-8c10-197c19f9a3aa" path="/var/lib/kubelet/pods/372e3319-752f-43bb-8c10-197c19f9a3aa/volumes"
Nov 02 17:49:44.412948 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:44.412915    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"
Nov 02 17:49:44.418955 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:44.418926    4456 scope.go:122] "RemoveContainer" containerID="f566147f7e53e3e16ae2bc12281b8689cc7650d8a1cb88c8af681363a7b9544d"
Nov 02 17:49:44.432494 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:44.432467    4456 scope.go:122] "RemoveContainer" containerID="4db660d01066fd539442dd27fc279e629403a01712229dc39f65b23ded3509d9"
Nov 02 17:49:44.445422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:44.445398    4456 scope.go:122] "RemoveContainer" containerID="a2575df33c8bc59b1996d3ad98250e29beef5e1a5d7ad2610ca1f7306edee51a"
Nov 02 17:49:44.459726 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:44.459678    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd" podStartSLOduration=3.459662992 podStartE2EDuration="3.459662992s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:43.62613959 +0000 UTC m=+1200.117202752" watchObservedRunningTime="2026-11-02 17:49:44.459662992 +0000 UTC m=+1200.950726154"
Nov 02 17:49:44.526172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:44.524603    4456 scope.go:122] "RemoveContainer" containerID="eea951ab2e361567d81c71f7178d26251ee9b619b9390993938bf8c8e5960c8d"
Nov 02 17:49:44.591849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:44.591707    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:44.648049 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:44.638439    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:44.657726 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:44.655399    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:44.657726 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:44.656916    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:49:46.130623 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:46.130575    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:46.168121 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:49:46.168089    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:49:46.308982 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:46.308952    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:46.482877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:46.476152    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:47.941786 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:47.941755    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:48.057365 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.057319    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"]
Nov 02 17:49:48.153608 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.153572    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:48.154319 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.154179    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"]
Nov 02 17:49:48.154528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.154410    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"
Nov 02 17:49:48.174847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.160083    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"]
Nov 02 17:49:48.195893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.191364    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-a-786c42\""
Nov 02 17:49:48.274874 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.234416    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-a-786c42-dockercfg-f2rpv\""
Nov 02 17:49:48.346887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.339387    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"
Nov 02 17:49:48.346887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.343773    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"]
Nov 02 17:49:48.346887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.345693    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n\" (UID: \"4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"
Nov 02 17:49:48.346887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.346289    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pwk5p\" (UniqueName: \"kubernetes.io/projected/4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0-kube-api-access-pwk5p\") pod \"payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n\" (UID: \"4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"
Nov 02 17:49:48.450530 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.450446    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wkj5w\" (UniqueName: \"kubernetes.io/projected/d21bb8e7-fe0a-4654-8373-429d9a092116-kube-api-access-wkj5w\") pod \"payload-processing-e2e-api-a-786c42-7c6f577947-zpthp\" (UID: \"d21bb8e7-fe0a-4654-8373-429d9a092116\") " pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"
Nov 02 17:49:48.453429 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.450642    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pwk5p\" (UniqueName: \"kubernetes.io/projected/4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0-kube-api-access-pwk5p\") pod \"payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n\" (UID: \"4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"
Nov 02 17:49:48.453429 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.450883    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n\" (UID: \"4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"
Nov 02 17:49:48.453429 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.450982    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d21bb8e7-fe0a-4654-8373-429d9a092116-plugins-config-volume\") pod \"payload-processing-e2e-api-a-786c42-7c6f577947-zpthp\" (UID: \"d21bb8e7-fe0a-4654-8373-429d9a092116\") " pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"
Nov 02 17:49:48.453429 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.451762    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n\" (UID: \"4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"
Nov 02 17:49:48.476095 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.476069    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pwk5p\" (UniqueName: \"kubernetes.io/projected/4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0-kube-api-access-pwk5p\") pod \"payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n\" (UID: \"4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0\") " pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"
Nov 02 17:49:48.552878 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.552791    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d21bb8e7-fe0a-4654-8373-429d9a092116-plugins-config-volume\") pod \"payload-processing-e2e-api-a-786c42-7c6f577947-zpthp\" (UID: \"d21bb8e7-fe0a-4654-8373-429d9a092116\") " pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"
Nov 02 17:49:48.553080 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.552984    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wkj5w\" (UniqueName: \"kubernetes.io/projected/d21bb8e7-fe0a-4654-8373-429d9a092116-kube-api-access-wkj5w\") pod \"payload-processing-e2e-api-a-786c42-7c6f577947-zpthp\" (UID: \"d21bb8e7-fe0a-4654-8373-429d9a092116\") " pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"
Nov 02 17:49:48.553560 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.553534    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d21bb8e7-fe0a-4654-8373-429d9a092116-plugins-config-volume\") pod \"payload-processing-e2e-api-a-786c42-7c6f577947-zpthp\" (UID: \"d21bb8e7-fe0a-4654-8373-429d9a092116\") " pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"
Nov 02 17:49:48.579724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.579693    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wkj5w\" (UniqueName: \"kubernetes.io/projected/d21bb8e7-fe0a-4654-8373-429d9a092116-kube-api-access-wkj5w\") pod \"payload-processing-e2e-api-a-786c42-7c6f577947-zpthp\" (UID: \"d21bb8e7-fe0a-4654-8373-429d9a092116\") " pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"
Nov 02 17:49:48.739162 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.739088    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"
Nov 02 17:49:48.816307 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.816065    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"
Nov 02 17:49:48.845849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:48.842070    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:49.168403 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:49:49.168368    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod4b8c06ad_f34b_4b8a_9612_f52c5cf0d7e0.slice/crio-7a9436c8a3a24bc5320c7a3e4a058fd64f4a8d676700023576750b3f979f3d04 WatchSource:0}: Error finding container 7a9436c8a3a24bc5320c7a3e4a058fd64f4a8d676700023576750b3f979f3d04: Status 404 returned error can't find the container with id 7a9436c8a3a24bc5320c7a3e4a058fd64f4a8d676700023576750b3f979f3d04
Nov 02 17:49:49.169153 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:49.169132    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"]
Nov 02 17:49:49.245945 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:49:49.245891    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd21bb8e7_fe0a_4654_8373_429d9a092116.slice/crio-7db00591c9bf74d2a82a079a17a0ad707da82f46b7327cbdce9a58bc310cbdfd WatchSource:0}: Error finding container 7db00591c9bf74d2a82a079a17a0ad707da82f46b7327cbdce9a58bc310cbdfd: Status 404 returned error can't find the container with id 7db00591c9bf74d2a82a079a17a0ad707da82f46b7327cbdce9a58bc310cbdfd
Nov 02 17:49:49.247679 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:49.247654    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"]
Nov 02 17:49:49.703848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:49.703754    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp" event={"ID":"d21bb8e7-fe0a-4654-8373-429d9a092116","Type":"ContainerStarted","Data":"c6d7b6ab73f2b06d67ed3aff51f34f02e7c80933c15c7c07b1676c73dde51a26"}
Nov 02 17:49:49.704184 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:49.703887    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"
Nov 02 17:49:49.704184 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:49.703911    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp" event={"ID":"d21bb8e7-fe0a-4654-8373-429d9a092116","Type":"ContainerStarted","Data":"7db00591c9bf74d2a82a079a17a0ad707da82f46b7327cbdce9a58bc310cbdfd"}
Nov 02 17:49:49.705808 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:49.705776    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n" event={"ID":"4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0","Type":"ContainerStarted","Data":"9e18613b05c3c7712c524b41dc4770d93aa259fdf33c2b37ca3a8bbeb9111fda"}
Nov 02 17:49:49.706174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:49.705840    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n" event={"ID":"4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0","Type":"ContainerStarted","Data":"7a9436c8a3a24bc5320c7a3e4a058fd64f4a8d676700023576750b3f979f3d04"}
Nov 02 17:49:49.706578 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:49.706560    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"
Nov 02 17:49:49.748329 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:49.748281    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp" podStartSLOduration=1.748267186 podStartE2EDuration="1.748267186s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:49.745132713 +0000 UTC m=+1206.236195875" watchObservedRunningTime="2026-11-02 17:49:49.748267186 +0000 UTC m=+1206.239330344"
Nov 02 17:49:49.790351 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:49.790285    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n" podStartSLOduration=1.790265311 podStartE2EDuration="1.790265311s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:49:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:49:49.786369124 +0000 UTC m=+1206.277432323" watchObservedRunningTime="2026-11-02 17:49:49.790265311 +0000 UTC m=+1206.281328529"
Nov 02 17:49:50.629584 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:50.629546    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:51.192370 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:51.192333    4456 patch_prober.go:29] interesting pod/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:49:51.192572 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:51.192398    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6" podUID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:49:51.745476 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:51.745440    4456 generic.go:356] "Generic (PLEG): container finished" podID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" containerID="131e4bce0b28eec82743c5f5c4e6857572a311fdf264f6220cebc081de88ca3e" exitCode=0
Nov 02 17:49:51.745993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:51.745511    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6" event={"ID":"0e46fada-1bd0-4b22-96ab-3c49c4b3199e","Type":"ContainerDied","Data":"131e4bce0b28eec82743c5f5c4e6857572a311fdf264f6220cebc081de88ca3e"}
Nov 02 17:49:51.746080 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:51.746057    4456 scope.go:122] "RemoveContainer" containerID="131e4bce0b28eec82743c5f5c4e6857572a311fdf264f6220cebc081de88ca3e"
Nov 02 17:49:51.747889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:51.747858    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:51.748017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:51.747944    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:51.748017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:51.747990    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:49:52.049948 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:52.049863    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"]
Nov 02 17:49:52.057092 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:52.057062    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-827c31a3-hg95f"]
Nov 02 17:49:52.319633 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:52.319550    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:52.508842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:52.507881    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:52.681896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:52.676267    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:52.778455 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:52.778414    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6" event={"ID":"0e46fada-1bd0-4b22-96ab-3c49c4b3199e","Type":"ContainerStarted","Data":"b8e26c4e4edaaffcb5446aa0eb600b76142fc6e40a41d43c05253afd1e946b94"}
Nov 02 17:49:53.214879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:53.214797    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:49:53.215098 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:53.215074    4456 patch_prober.go:29] interesting pod/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 02 17:49:53.215162 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:53.215139    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6" podUID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 02 17:49:53.316549 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:53.316511    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:53.506440 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:53.506350    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:53.623636 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:53.623598    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="50522edc-3a5e-4c72-a147-8b99ba6c9dc1" path="/var/lib/kubelet/pods/50522edc-3a5e-4c72-a147-8b99ba6c9dc1/volumes"
Nov 02 17:49:54.218532 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:54.218488    4456 patch_prober.go:29] interesting pod/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 02 17:49:54.218981 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:54.218571    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6" podUID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 02 17:49:54.346528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:54.346492    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:55.048625 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:55.048585    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"]
Nov 02 17:49:55.059888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:55.059853    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f6eed92d-rnwjw"]
Nov 02 17:49:55.215015 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:55.214977    4456 patch_prober.go:29] interesting pod/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 02 17:49:55.215242 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:55.215039    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6" podUID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 02 17:49:55.621828 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:55.621790    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cd769fa3-1722-4b6d-85a2-88602431a9aa" path="/var/lib/kubelet/pods/cd769fa3-1722-4b6d-85a2-88602431a9aa/volumes"
Nov 02 17:49:56.066785 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:56.066694    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:56.215200 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:56.215137    4456 patch_prober.go:29] interesting pod/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 02 17:49:56.215434 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:56.215225    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6" podUID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 02 17:49:57.215256 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:57.215215    4456 patch_prober.go:29] interesting pod/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused" start-of-body=
Nov 02 17:49:57.215694 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:57.215275    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6" podUID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.127:15021/healthz/ready\": dial tcp 10.134.0.127:15021: connect: connection refused"
Nov 02 17:49:57.914232 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:57.914196    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:58.068848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:58.068175    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:49:58.088479 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:58.088448    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"]
Nov 02 17:49:58.099917 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:58.099877    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-28b398b4-9w4dl"]
Nov 02 17:49:58.219319 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:58.219289    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:49:58.219886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:58.219866    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:49:58.220677 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:58.220661    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:49:59.617314 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:59.617278    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="efa7b357-2411-42dd-837a-cc36d6133378" path="/var/lib/kubelet/pods/efa7b357-2411-42dd-837a-cc36d6133378/volumes"
Nov 02 17:49:59.899700 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:49:59.899608    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:00.039924 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:00.039884    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"]
Nov 02 17:50:00.049122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:00.049089    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e9ffd4bb-8jqvf"]
Nov 02 17:50:00.738537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:00.738512    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"
Nov 02 17:50:01.497739 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:01.497699    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:01.617773 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:01.617737    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="84a0333b-4b46-4853-af3c-a5aeb904a877" path="/var/lib/kubelet/pods/84a0333b-4b46-4853-af3c-a5aeb904a877/volumes"
Nov 02 17:50:01.744054 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:01.744017    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:01.754402 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:01.754314    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"
Nov 02 17:50:03.594801 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:03.594735    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:05.452426 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:05.452383    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:07.299248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:07.299186    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:09.207926 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:09.207889    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:09.387718 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:09.387681    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:10.141594 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:10.141560    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:10.314825 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:10.314767    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:10.809113 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:10.809079    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:11.158557 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:11.158448    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:11.849149 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:11.849118    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:12.328183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:12.328144    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:12.956395 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:12.956355    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:13.310544 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:13.310404    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:14.973138 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:14.973099    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:16.167544 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:50:16.167504    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:50:17.048721 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:17.048623    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:17.066997 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:17.066965    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"]
Nov 02 17:50:17.079008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:17.078968    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-671ad5c9-gjvpj"]
Nov 02 17:50:17.269155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:17.269110    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:17.334343 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:17.329445    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"]
Nov 02 17:50:17.334343 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:17.329724    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t" podUID="17f59d36-8a89-4385-bdaf-40777f5e1a4e" containerName="istio-proxy" containerID="cri-o://349f7a8f4f162e2da516d281b9f4438830f8645cf63394b83ab66e7b973dc19a" gracePeriod=30
Nov 02 17:50:17.429195 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:17.419044    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"]
Nov 02 17:50:17.429195 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:17.419326    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr" podUID="fb933398-5485-4600-a38e-640531c54137" containerName="istio-proxy" containerID="cri-o://c87a2115f023acd478f1d6c28935f520f81e705ba4d79b759e5802589ddebe1c" gracePeriod=30
Nov 02 17:50:17.491136 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:17.468864    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"]
Nov 02 17:50:17.491136 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:17.469118    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf" podUID="babcae9b-991f-4b7b-8d1c-7a4d7804bc24" containerName="istio-proxy" containerID="cri-o://eab8c74e617923c2b748a1867f5951dac5437af3dd2db8e5ecabf5afa94bb225" gracePeriod=30
Nov 02 17:50:17.491136 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:17.479020    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"]
Nov 02 17:50:17.491136 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:17.479246    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq" podUID="c1d9a093-015c-498b-8808-7939c01499da" containerName="istio-proxy" containerID="cri-o://dac0a2352ac1baae2810fea9acc9b8c90d9dee012f5df6f86ddb0a56f3c33341" gracePeriod=30
Nov 02 17:50:17.679446 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:17.679303    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d861846c-2815-4acf-8d17-ef905e132572" path="/var/lib/kubelet/pods/d861846c-2815-4acf-8d17-ef905e132572/volumes"
Nov 02 17:50:19.448173 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:19.446435    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:19.869891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:19.858980    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:20.294647 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:20.294329    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:20.475219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:20.475187    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:21.642759 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:21.642692    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:21.839206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:21.839169    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:23.694630 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:23.693379    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:23.741102 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:23.740619    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:50:23.849009 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:23.848968    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:50:23.918846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:23.912453    4456 generic.go:356] "Generic (PLEG): container finished" podID="babcae9b-991f-4b7b-8d1c-7a4d7804bc24" containerID="eab8c74e617923c2b748a1867f5951dac5437af3dd2db8e5ecabf5afa94bb225" exitCode=0
Nov 02 17:50:23.918846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:23.912603    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf" event={"ID":"babcae9b-991f-4b7b-8d1c-7a4d7804bc24","Type":"ContainerDied","Data":"eab8c74e617923c2b748a1867f5951dac5437af3dd2db8e5ecabf5afa94bb225"}
Nov 02 17:50:23.969169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:23.969091    4456 generic.go:356] "Generic (PLEG): container finished" podID="c1d9a093-015c-498b-8808-7939c01499da" containerID="dac0a2352ac1baae2810fea9acc9b8c90d9dee012f5df6f86ddb0a56f3c33341" exitCode=0
Nov 02 17:50:23.969525 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:23.969165    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq" event={"ID":"c1d9a093-015c-498b-8808-7939c01499da","Type":"ContainerDied","Data":"dac0a2352ac1baae2810fea9acc9b8c90d9dee012f5df6f86ddb0a56f3c33341"}
Nov 02 17:50:23.970832 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:23.970792    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:50:24.002690 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.002651    4456 generic.go:356] "Generic (PLEG): container finished" podID="fb933398-5485-4600-a38e-640531c54137" containerID="c87a2115f023acd478f1d6c28935f520f81e705ba4d79b759e5802589ddebe1c" exitCode=0
Nov 02 17:50:24.003158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.003129    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr" event={"ID":"fb933398-5485-4600-a38e-640531c54137","Type":"ContainerDied","Data":"c87a2115f023acd478f1d6c28935f520f81e705ba4d79b759e5802589ddebe1c"}
Nov 02 17:50:24.057841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.051983    4456 scope.go:122] "RemoveContainer" containerID="c87a2115f023acd478f1d6c28935f520f81e705ba4d79b759e5802589ddebe1c"
Nov 02 17:50:24.058434 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.058403    4456 generic.go:356] "Generic (PLEG): container finished" podID="17f59d36-8a89-4385-bdaf-40777f5e1a4e" containerID="349f7a8f4f162e2da516d281b9f4438830f8645cf63394b83ab66e7b973dc19a" exitCode=0
Nov 02 17:50:24.058698 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.058486    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t" event={"ID":"17f59d36-8a89-4385-bdaf-40777f5e1a4e","Type":"ContainerDied","Data":"349f7a8f4f162e2da516d281b9f4438830f8645cf63394b83ab66e7b973dc19a"}
Nov 02 17:50:24.077109 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.077020    4456 scope.go:122] "RemoveContainer" containerID="efc584ce44ecf1a685d22ebf1c76e1b3706e98d16b83773fa74439aab3a7eb04"
Nov 02 17:50:24.170889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.164204    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-workload-socket\") pod \"fb933398-5485-4600-a38e-640531c54137\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") "
Nov 02 17:50:24.170889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.164338    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-istio-data\") pod \"fb933398-5485-4600-a38e-640531c54137\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") "
Nov 02 17:50:24.170889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.164458    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb933398-5485-4600-a38e-640531c54137-kube-api-access-t994b\" (UniqueName: \"kubernetes.io/projected/fb933398-5485-4600-a38e-640531c54137-kube-api-access-t994b\") pod \"fb933398-5485-4600-a38e-640531c54137\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") "
Nov 02 17:50:24.170889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.164498    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-workload-certs\") pod \"fb933398-5485-4600-a38e-640531c54137\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") "
Nov 02 17:50:24.170889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.164531    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fb933398-5485-4600-a38e-640531c54137-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb933398-5485-4600-a38e-640531c54137-istiod-ca-cert\") pod \"fb933398-5485-4600-a38e-640531c54137\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") "
Nov 02 17:50:24.170889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.164645    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-credential-socket\") pod \"fb933398-5485-4600-a38e-640531c54137\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") "
Nov 02 17:50:24.170889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.164713    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fb933398-5485-4600-a38e-640531c54137-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb933398-5485-4600-a38e-640531c54137-istio-podinfo\") pod \"fb933398-5485-4600-a38e-640531c54137\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") "
Nov 02 17:50:24.170889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.164778    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fb933398-5485-4600-a38e-640531c54137-istio-token\" (UniqueName: \"kubernetes.io/projected/fb933398-5485-4600-a38e-640531c54137-istio-token\") pod \"fb933398-5485-4600-a38e-640531c54137\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") "
Nov 02 17:50:24.170889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.164872    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-istio-envoy\") pod \"fb933398-5485-4600-a38e-640531c54137\" (UID: \"fb933398-5485-4600-a38e-640531c54137\") "
Nov 02 17:50:24.170889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.168524    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fb933398-5485-4600-a38e-640531c54137-istiod-ca-cert" pod "fb933398-5485-4600-a38e-640531c54137" (UID: "fb933398-5485-4600-a38e-640531c54137"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:24.170889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.168778    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-credential-socket" pod "fb933398-5485-4600-a38e-640531c54137" (UID: "fb933398-5485-4600-a38e-640531c54137"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.172147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.172112    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-workload-socket" pod "fb933398-5485-4600-a38e-640531c54137" (UID: "fb933398-5485-4600-a38e-640531c54137"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.188017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.183900    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-istio-data" pod "fb933398-5485-4600-a38e-640531c54137" (UID: "fb933398-5485-4600-a38e-640531c54137"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.188017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.183989    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-workload-certs" pod "fb933398-5485-4600-a38e-640531c54137" (UID: "fb933398-5485-4600-a38e-640531c54137"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.203446 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.193971    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-istio-envoy" pod "fb933398-5485-4600-a38e-640531c54137" (UID: "fb933398-5485-4600-a38e-640531c54137"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.270870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.270081    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.270870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.270114    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.270870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.270127    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.270870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.270159    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fb933398-5485-4600-a38e-640531c54137-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.270870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.270174    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.270870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.270189    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fb933398-5485-4600-a38e-640531c54137-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.360840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.358797    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:50:24.362010 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.361878    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fb933398-5485-4600-a38e-640531c54137-istio-podinfo" pod "fb933398-5485-4600-a38e-640531c54137" (UID: "fb933398-5485-4600-a38e-640531c54137"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:50:24.362352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.362308    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb933398-5485-4600-a38e-640531c54137-kube-api-access-t994b" pod "fb933398-5485-4600-a38e-640531c54137" (UID: "fb933398-5485-4600-a38e-640531c54137"). InnerVolumeSpecName "kube-api-access-t994b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:24.364228 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.363997    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:50:24.365897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.365203    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fb933398-5485-4600-a38e-640531c54137-istio-token" pod "fb933398-5485-4600-a38e-640531c54137" (UID: "fb933398-5485-4600-a38e-640531c54137"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.373533    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-token\" (UniqueName: \"kubernetes.io/projected/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-token\") pod \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.373574    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-podinfo\") pod \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.373656    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-credential-socket\") pod \"c1d9a093-015c-498b-8808-7939c01499da\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.373686    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c1d9a093-015c-498b-8808-7939c01499da-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d9a093-015c-498b-8808-7939c01499da-istio-podinfo\") pod \"c1d9a093-015c-498b-8808-7939c01499da\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.373716    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-credential-socket\") pod \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.373747    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c1d9a093-015c-498b-8808-7939c01499da-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d9a093-015c-498b-8808-7939c01499da-istiod-ca-cert\") pod \"c1d9a093-015c-498b-8808-7939c01499da\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.373782    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d9a093-015c-498b-8808-7939c01499da-kube-api-access-7zvs6\" (UniqueName: \"kubernetes.io/projected/c1d9a093-015c-498b-8808-7939c01499da-kube-api-access-7zvs6\") pod \"c1d9a093-015c-498b-8808-7939c01499da\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.373831    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-data\") pod \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.373927    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-workload-certs\") pod \"c1d9a093-015c-498b-8808-7939c01499da\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.374007    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-workload-certs\") pod \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.374133    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-workload-socket\") pod \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.374356    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-istio-envoy\") pod \"c1d9a093-015c-498b-8808-7939c01499da\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.374389    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c1d9a093-015c-498b-8808-7939c01499da-istio-token\" (UniqueName: \"kubernetes.io/projected/c1d9a093-015c-498b-8808-7939c01499da-istio-token\") pod \"c1d9a093-015c-498b-8808-7939c01499da\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.374471    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-envoy\") pod \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.374517    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-workload-socket\") pod \"c1d9a093-015c-498b-8808-7939c01499da\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.374563    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-kube-api-access-nq9jv\" (UniqueName: \"kubernetes.io/projected/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-kube-api-access-nq9jv\") pod \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.374605    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-istio-data\") pod \"c1d9a093-015c-498b-8808-7939c01499da\" (UID: \"c1d9a093-015c-498b-8808-7939c01499da\") "
Nov 02 17:50:24.374836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.374652    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istiod-ca-cert\") pod \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\" (UID: \"babcae9b-991f-4b7b-8d1c-7a4d7804bc24\") "
Nov 02 17:50:24.377109 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.376853    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t994b\" (UniqueName: \"kubernetes.io/projected/fb933398-5485-4600-a38e-640531c54137-kube-api-access-t994b\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.377109 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.376883    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fb933398-5485-4600-a38e-640531c54137-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.377109 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.376898    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fb933398-5485-4600-a38e-640531c54137-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.377754 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.377426    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istiod-ca-cert" pod "babcae9b-991f-4b7b-8d1c-7a4d7804bc24" (UID: "babcae9b-991f-4b7b-8d1c-7a4d7804bc24"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:24.378456 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.378435    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-workload-certs" pod "c1d9a093-015c-498b-8808-7939c01499da" (UID: "c1d9a093-015c-498b-8808-7939c01499da"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.378977 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.378673    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-workload-socket" pod "c1d9a093-015c-498b-8808-7939c01499da" (UID: "c1d9a093-015c-498b-8808-7939c01499da"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.378977 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.378889    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c1d9a093-015c-498b-8808-7939c01499da-istiod-ca-cert" pod "c1d9a093-015c-498b-8808-7939c01499da" (UID: "c1d9a093-015c-498b-8808-7939c01499da"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:24.379194 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.379169    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-credential-socket" pod "babcae9b-991f-4b7b-8d1c-7a4d7804bc24" (UID: "babcae9b-991f-4b7b-8d1c-7a4d7804bc24"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.396266 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.379512    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-data" pod "babcae9b-991f-4b7b-8d1c-7a4d7804bc24" (UID: "babcae9b-991f-4b7b-8d1c-7a4d7804bc24"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.396266 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.379696    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-credential-socket" pod "c1d9a093-015c-498b-8808-7939c01499da" (UID: "c1d9a093-015c-498b-8808-7939c01499da"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.396266 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.386017    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-istio-data" pod "c1d9a093-015c-498b-8808-7939c01499da" (UID: "c1d9a093-015c-498b-8808-7939c01499da"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.396266 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.386199    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-workload-socket" pod "babcae9b-991f-4b7b-8d1c-7a4d7804bc24" (UID: "babcae9b-991f-4b7b-8d1c-7a4d7804bc24"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.396266 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.395913    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-workload-certs" pod "babcae9b-991f-4b7b-8d1c-7a4d7804bc24" (UID: "babcae9b-991f-4b7b-8d1c-7a4d7804bc24"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.418857 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.416800    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-podinfo" pod "babcae9b-991f-4b7b-8d1c-7a4d7804bc24" (UID: "babcae9b-991f-4b7b-8d1c-7a4d7804bc24"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:50:24.449958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.449904    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-token" pod "babcae9b-991f-4b7b-8d1c-7a4d7804bc24" (UID: "babcae9b-991f-4b7b-8d1c-7a4d7804bc24"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:24.453756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.453709    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-envoy" pod "babcae9b-991f-4b7b-8d1c-7a4d7804bc24" (UID: "babcae9b-991f-4b7b-8d1c-7a4d7804bc24"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.453756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.453727    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-istio-envoy" pod "c1d9a093-015c-498b-8808-7939c01499da" (UID: "c1d9a093-015c-498b-8808-7939c01499da"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.454212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.454159    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d9a093-015c-498b-8808-7939c01499da-kube-api-access-7zvs6" pod "c1d9a093-015c-498b-8808-7939c01499da" (UID: "c1d9a093-015c-498b-8808-7939c01499da"). InnerVolumeSpecName "kube-api-access-7zvs6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:24.454322 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.454243    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c1d9a093-015c-498b-8808-7939c01499da-istio-podinfo" pod "c1d9a093-015c-498b-8808-7939c01499da" (UID: "c1d9a093-015c-498b-8808-7939c01499da"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:50:24.454490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.454465    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c1d9a093-015c-498b-8808-7939c01499da-istio-token" pod "c1d9a093-015c-498b-8808-7939c01499da" (UID: "c1d9a093-015c-498b-8808-7939c01499da"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:24.455073 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.455053    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:50:24.456311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.456292    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-kube-api-access-nq9jv" pod "babcae9b-991f-4b7b-8d1c-7a4d7804bc24" (UID: "babcae9b-991f-4b7b-8d1c-7a4d7804bc24"). InnerVolumeSpecName "kube-api-access-nq9jv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:24.479398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.478389    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-podinfo\") pod \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") "
Nov 02 17:50:24.479398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.478437    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-credential-socket\") pod \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") "
Nov 02 17:50:24.479398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.478487    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f59d36-8a89-4385-bdaf-40777f5e1a4e-kube-api-access-t42n8\" (UniqueName: \"kubernetes.io/projected/17f59d36-8a89-4385-bdaf-40777f5e1a4e-kube-api-access-t42n8\") pod \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") "
Nov 02 17:50:24.479398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.478552    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istiod-ca-cert\") pod \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") "
Nov 02 17:50:24.479398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.478633    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-workload-certs\") pod \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") "
Nov 02 17:50:24.479398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.478665    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-workload-socket\") pod \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") "
Nov 02 17:50:24.479398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.478699    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-data\") pod \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") "
Nov 02 17:50:24.479398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.478714    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-envoy\") pod \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") "
Nov 02 17:50:24.479398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.478835    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-token\" (UniqueName: \"kubernetes.io/projected/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-token\") pod \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\" (UID: \"17f59d36-8a89-4385-bdaf-40777f5e1a4e\") "
Nov 02 17:50:24.479398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.479045    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-workload-socket" pod "17f59d36-8a89-4385-bdaf-40777f5e1a4e" (UID: "17f59d36-8a89-4385-bdaf-40777f5e1a4e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.479398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.479118    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-workload-certs" pod "17f59d36-8a89-4385-bdaf-40777f5e1a4e" (UID: "17f59d36-8a89-4385-bdaf-40777f5e1a4e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.479398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.479163    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istiod-ca-cert" pod "17f59d36-8a89-4385-bdaf-40777f5e1a4e" (UID: "17f59d36-8a89-4385-bdaf-40777f5e1a4e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:24.479398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.479194    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-credential-socket" pod "17f59d36-8a89-4385-bdaf-40777f5e1a4e" (UID: "17f59d36-8a89-4385-bdaf-40777f5e1a4e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.492908    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-data" pod "17f59d36-8a89-4385-bdaf-40777f5e1a4e" (UID: "17f59d36-8a89-4385-bdaf-40777f5e1a4e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.501937    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.501967    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.502001    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c1d9a093-015c-498b-8808-7939c01499da-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.502014    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.502028    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.502041    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.502069    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nq9jv\" (UniqueName: \"kubernetes.io/projected/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-kube-api-access-nq9jv\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.502085    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.502098    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.502111    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.502123    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.502151    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.502164    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.502178    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c1d9a093-015c-498b-8808-7939c01499da-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.513859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.502191    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.544016 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.515053    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c1d9a093-015c-498b-8808-7939c01499da-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.544016 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.515115    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7zvs6\" (UniqueName: \"kubernetes.io/projected/c1d9a093-015c-498b-8808-7939c01499da-kube-api-access-7zvs6\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.544016 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.515132    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.544016 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.515163    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.544016 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.515178    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c1d9a093-015c-498b-8808-7939c01499da-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.544016 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.515191    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.544016 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.515206    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/babcae9b-991f-4b7b-8d1c-7a4d7804bc24-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.556226 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.554998    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-podinfo" pod "17f59d36-8a89-4385-bdaf-40777f5e1a4e" (UID: "17f59d36-8a89-4385-bdaf-40777f5e1a4e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:50:24.556600 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.556560    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-token" pod "17f59d36-8a89-4385-bdaf-40777f5e1a4e" (UID: "17f59d36-8a89-4385-bdaf-40777f5e1a4e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:24.556972 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.556597    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/17f59d36-8a89-4385-bdaf-40777f5e1a4e-kube-api-access-t42n8" pod "17f59d36-8a89-4385-bdaf-40777f5e1a4e" (UID: "17f59d36-8a89-4385-bdaf-40777f5e1a4e"). InnerVolumeSpecName "kube-api-access-t42n8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:24.562669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.562615    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-envoy" pod "17f59d36-8a89-4385-bdaf-40777f5e1a4e" (UID: "17f59d36-8a89-4385-bdaf-40777f5e1a4e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:24.639838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.639422    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.639838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.639459    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.639838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.639474    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.639838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.639487    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/17f59d36-8a89-4385-bdaf-40777f5e1a4e-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:24.639838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:24.639501    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t42n8\" (UniqueName: \"kubernetes.io/projected/17f59d36-8a89-4385-bdaf-40777f5e1a4e-kube-api-access-t42n8\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:25.071583 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.071537    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf" event={"ID":"babcae9b-991f-4b7b-8d1c-7a4d7804bc24","Type":"ContainerDied","Data":"45cf2750410bf31298fd5b6917007f6227a74f3f418d47ac51390880d71a2a83"}
Nov 02 17:50:25.071994 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.071548    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"
Nov 02 17:50:25.071994 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.071638    4456 scope.go:122] "RemoveContainer" containerID="eab8c74e617923c2b748a1867f5951dac5437af3dd2db8e5ecabf5afa94bb225"
Nov 02 17:50:25.073459 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.073438    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"
Nov 02 17:50:25.073567 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.073461    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq" event={"ID":"c1d9a093-015c-498b-8808-7939c01499da","Type":"ContainerDied","Data":"fd00914ee78a8ce938a74edc9a6ab41beddba2384c41c6b7134e9b0e33273c0c"}
Nov 02 17:50:25.075248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.075216    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"
Nov 02 17:50:25.075354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.075259    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr" event={"ID":"fb933398-5485-4600-a38e-640531c54137","Type":"ContainerDied","Data":"ec06b06c2158e74a662d63cf5e79c65b848abe3e82ccc2b7339d79f5257d9e96"}
Nov 02 17:50:25.076944 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.076923    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"
Nov 02 17:50:25.077033 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.076970    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t" event={"ID":"17f59d36-8a89-4385-bdaf-40777f5e1a4e","Type":"ContainerDied","Data":"d0537c97f868c7de7a7043431bfb11b2334963de56e3b55028a8fe9d138b59f5"}
Nov 02 17:50:25.129102 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.127290    4456 scope.go:122] "RemoveContainer" containerID="dac0a2352ac1baae2810fea9acc9b8c90d9dee012f5df6f86ddb0a56f3c33341"
Nov 02 17:50:25.140647 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.140622    4456 scope.go:122] "RemoveContainer" containerID="349f7a8f4f162e2da516d281b9f4438830f8645cf63394b83ab66e7b973dc19a"
Nov 02 17:50:25.226838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.225854    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"]
Nov 02 17:50:25.236618 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.236582    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-ld45t"]
Nov 02 17:50:25.262494 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.262458    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"]
Nov 02 17:50:25.268969 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.268941    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-nmhmr"]
Nov 02 17:50:25.309923 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.309888    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"]
Nov 02 17:50:25.315801 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.315767    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-hg6pf"]
Nov 02 17:50:25.353353 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.352898    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"]
Nov 02 17:50:25.359569 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.359542    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w2-f0a3b8-openshift-default-54cd4fbd6f-9grfq"]
Nov 02 17:50:25.502849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.489507    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:25.623613 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.623576    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="17f59d36-8a89-4385-bdaf-40777f5e1a4e" path="/var/lib/kubelet/pods/17f59d36-8a89-4385-bdaf-40777f5e1a4e/volumes"
Nov 02 17:50:25.624392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.624369    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="babcae9b-991f-4b7b-8d1c-7a4d7804bc24" path="/var/lib/kubelet/pods/babcae9b-991f-4b7b-8d1c-7a4d7804bc24/volumes"
Nov 02 17:50:25.625113 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.625061    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c1d9a093-015c-498b-8808-7939c01499da" path="/var/lib/kubelet/pods/c1d9a093-015c-498b-8808-7939c01499da/volumes"
Nov 02 17:50:25.627633 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:25.626130    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fb933398-5485-4600-a38e-640531c54137" path="/var/lib/kubelet/pods/fb933398-5485-4600-a38e-640531c54137/volumes"
Nov 02 17:50:26.904061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:26.904020    4456 patch_prober.go:29] interesting pod/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:50:26.904567 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:26.904104    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4" podUID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:50:27.338846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:27.334717    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:27.495514 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:27.495191    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:29.233714 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:29.233675    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:31.066336 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:31.066298    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"]
Nov 02 17:50:31.078789 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:31.078758    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53683142-bmtlf"]
Nov 02 17:50:31.125519 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:31.125040    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:31.619341 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:31.619303    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="736a6511-6750-49a8-8830-05c9dccd495c" path="/var/lib/kubelet/pods/736a6511-6750-49a8-8830-05c9dccd495c/volumes"
Nov 02 17:50:32.876551 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:32.875641    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:34.707605 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:34.707535    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:34.850760 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:34.850716    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:36.213609 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:36.213569    4456 generic.go:356] "Generic (PLEG): container finished" podID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerID="32c5e815dd3a88821539f897b9d64cab531b39e5121bc42aa66691c79ec0ae1b" exitCode=0
Nov 02 17:50:36.213609 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:36.213603    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4" event={"ID":"2268303a-b8a6-4783-b4fc-3467cd0c2bb3","Type":"ContainerDied","Data":"32c5e815dd3a88821539f897b9d64cab531b39e5121bc42aa66691c79ec0ae1b"}
Nov 02 17:50:36.214251 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:36.214203    4456 scope.go:122] "RemoveContainer" containerID="32c5e815dd3a88821539f897b9d64cab531b39e5121bc42aa66691c79ec0ae1b"
Nov 02 17:50:36.216124 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:36.216095    4456 generic.go:356] "Generic (PLEG): container finished" podID="98c10ff3-d496-4336-a2cd-921042c719d3" containerID="4c2b669d919f440b588554d96ba166ca71127cc37440ec0db956fe5315cc5fdc" exitCode=0
Nov 02 17:50:36.216265 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:36.216166    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7" event={"ID":"98c10ff3-d496-4336-a2cd-921042c719d3","Type":"ContainerDied","Data":"4c2b669d919f440b588554d96ba166ca71127cc37440ec0db956fe5315cc5fdc"}
Nov 02 17:50:36.216632 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:36.216614    4456 scope.go:122] "RemoveContainer" containerID="4c2b669d919f440b588554d96ba166ca71127cc37440ec0db956fe5315cc5fdc"
Nov 02 17:50:36.216734 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:36.216697    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:50:36.216801 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:36.216782    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:50:36.216877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:36.216859    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:50:36.218506 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:36.218480    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:50:36.218592 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:36.218548    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:50:36.218592 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:36.218582    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:50:36.767423 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:36.767316    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:37.241407 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:37.241368    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7" event={"ID":"98c10ff3-d496-4336-a2cd-921042c719d3","Type":"ContainerStarted","Data":"77600a881963b6c995f4d2afec219dad9a8788b39af4a147947ed27a989fad54"}
Nov 02 17:50:37.244224 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:37.244200    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4" event={"ID":"2268303a-b8a6-4783-b4fc-3467cd0c2bb3","Type":"ContainerStarted","Data":"24ae40e543016679f95f4771c3038102f64f7638708d8e314fe7c78ac01479fc"}
Nov 02 17:50:37.469209 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:37.469166    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:37.739744 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:37.739702    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:50:37.740992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:37.740965    4456 patch_prober.go:29] interesting pod/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 17:50:37.741122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:37.741010    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7" podUID="98c10ff3-d496-4336-a2cd-921042c719d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 17:50:38.165886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.165770    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:38.226188 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.226157    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:50:38.228956 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.228925    4456 patch_prober.go:29] interesting pod/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 17:50:38.229160 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.229135    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4" podUID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.610804    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"]
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.635142    4456 cpu_manager.go:414] "containerMap: removing container" podUID="fb933398-5485-4600-a38e-640531c54137" containerName="istio-proxy"
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.635164    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb933398-5485-4600-a38e-640531c54137" containerName="istio-proxy"
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.635219    4456 cpu_manager.go:414] "containerMap: removing container" podUID="fb933398-5485-4600-a38e-640531c54137" containerName="istio-proxy"
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.635229    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fb933398-5485-4600-a38e-640531c54137" containerName="istio-proxy"
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.635250    4456 cpu_manager.go:414] "containerMap: removing container" podUID="babcae9b-991f-4b7b-8d1c-7a4d7804bc24" containerName="istio-proxy"
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.635260    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="babcae9b-991f-4b7b-8d1c-7a4d7804bc24" containerName="istio-proxy"
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.635278    4456 cpu_manager.go:414] "containerMap: removing container" podUID="17f59d36-8a89-4385-bdaf-40777f5e1a4e" containerName="istio-proxy"
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.635285    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="17f59d36-8a89-4385-bdaf-40777f5e1a4e" containerName="istio-proxy"
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.635302    4456 cpu_manager.go:414] "containerMap: removing container" podUID="c1d9a093-015c-498b-8808-7939c01499da" containerName="istio-proxy"
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.635311    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c1d9a093-015c-498b-8808-7939c01499da" containerName="istio-proxy"
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.635514    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb933398-5485-4600-a38e-640531c54137" containerName="istio-proxy"
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.635528    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="babcae9b-991f-4b7b-8d1c-7a4d7804bc24" containerName="istio-proxy"
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.635541    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="17f59d36-8a89-4385-bdaf-40777f5e1a4e" containerName="istio-proxy"
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.635554    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="c1d9a093-015c-498b-8808-7939c01499da" containerName="istio-proxy"
Nov 02 17:50:38.637837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.635571    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="fb933398-5485-4600-a38e-640531c54137" containerName="istio-proxy"
Nov 02 17:50:38.692839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.677153    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:38.716755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.694162    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-769b70-openshift-default-dockercfg-fkpn8\""
Nov 02 17:50:38.716755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.697229    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"]
Nov 02 17:50:38.716755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.715592    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:38.740696 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.740269    4456 patch_prober.go:29] interesting pod/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 17:50:38.740696 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.740336    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7" podUID="98c10ff3-d496-4336-a2cd-921042c719d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 17:50:38.824449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.785651    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-workload-certs\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:38.824449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.785738    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a4895f-f62c-4012-8114-49a4e97a8644-istio-podinfo\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:38.824449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.785786    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a4895f-f62c-4012-8114-49a4e97a8644-istiod-ca-cert\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:38.824449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.785946    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-credential-socket\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:38.824449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.786015    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-istio-envoy\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:38.824449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.786057    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-workload-socket\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:38.824449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.786131    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a4895f-f62c-4012-8114-49a4e97a8644-istio-token\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:38.824449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.786163    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-istio-data\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:38.824449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:38.786307    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pfb5r\" (UniqueName: \"kubernetes.io/projected/92a4895f-f62c-4012-8114-49a4e97a8644-kube-api-access-pfb5r\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.071800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.071760    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pfb5r\" (UniqueName: \"kubernetes.io/projected/92a4895f-f62c-4012-8114-49a4e97a8644-kube-api-access-pfb5r\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.072184 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.072160    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-workload-certs\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.072412 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.072395    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a4895f-f62c-4012-8114-49a4e97a8644-istio-podinfo\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.072595 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.072569    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a4895f-f62c-4012-8114-49a4e97a8644-istiod-ca-cert\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.074018 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.073976    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-credential-socket\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.079023 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.078990    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-istio-envoy\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.079239 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.079204    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-workload-socket\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.079532 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.079486    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a4895f-f62c-4012-8114-49a4e97a8644-istio-token\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.079688 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.079671    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-istio-data\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.108876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.083445    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-workload-certs\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.135843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.126449    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-workload-socket\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.153309 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.138152    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a4895f-f62c-4012-8114-49a4e97a8644-istiod-ca-cert\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.153595 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.151889    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:39.155999 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.155151    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-credential-socket\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.156878 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.156854    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-istio-data\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.160274 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.160233    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a4895f-f62c-4012-8114-49a4e97a8644-istio-podinfo\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.160730 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.160671    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-istio-envoy\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.193993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.193962    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pfb5r\" (UniqueName: \"kubernetes.io/projected/92a4895f-f62c-4012-8114-49a4e97a8644-kube-api-access-pfb5r\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.195138 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.195089    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a4895f-f62c-4012-8114-49a4e97a8644-istio-token\") pod \"e2e-api-b-769b70-openshift-default-897676bcc-gfczr\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") " pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.243049 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.242955    4456 patch_prober.go:29] interesting pod/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 17:50:39.243049 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.243023    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4" podUID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 17:50:39.256849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.245349    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:50:39.344246 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.335352    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:39.369842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.356714    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:39.740700 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.740112    4456 patch_prober.go:29] interesting pod/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 17:50:39.740700 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.740174    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7" podUID="98c10ff3-d496-4336-a2cd-921042c719d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 17:50:39.879637 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:50:39.879120    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92a4895f_f62c_4012_8114_49a4e97a8644.slice/crio-0145904e38a853dddb60f8d9842215f9d906cf838e03a26d5a9585358a0e2d85 WatchSource:0}: Error finding container 0145904e38a853dddb60f8d9842215f9d906cf838e03a26d5a9585358a0e2d85: Status 404 returned error can't find the container with id 0145904e38a853dddb60f8d9842215f9d906cf838e03a26d5a9585358a0e2d85
Nov 02 17:50:39.896839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.880999    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"]
Nov 02 17:50:39.927842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.909353    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:50:39.927842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.909459    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:50:39.927842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:39.909507    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:50:40.228842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:40.221196    4456 patch_prober.go:29] interesting pod/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 17:50:40.228842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:40.221261    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4" podUID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 17:50:40.405843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:40.399032    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr" event={"ID":"92a4895f-f62c-4012-8114-49a4e97a8644","Type":"ContainerStarted","Data":"573cf1c3514bb6d994f4e99ee4466848b20f08591ecb7d4646a422ba5b8684aa"}
Nov 02 17:50:40.405843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:40.399098    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr" event={"ID":"92a4895f-f62c-4012-8114-49a4e97a8644","Type":"ContainerStarted","Data":"0145904e38a853dddb60f8d9842215f9d906cf838e03a26d5a9585358a0e2d85"}
Nov 02 17:50:40.459841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:40.458977    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr" podStartSLOduration=2.458953719 podStartE2EDuration="2.458953719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:40.452541185 +0000 UTC m=+1256.943604337" watchObservedRunningTime="2026-11-02 17:50:40.458953719 +0000 UTC m=+1256.950016912"
Nov 02 17:50:40.700444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:40.700392    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:40.742290 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:40.742251    4456 patch_prober.go:29] interesting pod/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 17:50:40.742919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:40.742875    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7" podUID="98c10ff3-d496-4336-a2cd-921042c719d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 17:50:40.872837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:40.869728    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:50:41.226842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:41.226372    4456 patch_prober.go:29] interesting pod/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 17:50:41.226842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:41.226452    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4" podUID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 17:50:41.357271 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:41.357235    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:41.363292 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:41.363262    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:41.451861 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:41.449416    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:41.451861 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:41.450890    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:50:41.747310 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:41.741977    4456 patch_prober.go:29] interesting pod/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 17:50:41.747310 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:41.742059    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7" podUID="98c10ff3-d496-4336-a2cd-921042c719d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 17:50:42.220839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:42.220193    4456 patch_prober.go:29] interesting pod/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 17:50:42.220839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:42.220259    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4" podUID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 17:50:42.732840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:42.732169    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:42.748841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:42.741256    4456 patch_prober.go:29] interesting pod/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused" start-of-body=
Nov 02 17:50:42.748841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:42.741317    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7" podUID="98c10ff3-d496-4336-a2cd-921042c719d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": dial tcp 10.134.0.136:15021: connect: connection refused"
Nov 02 17:50:43.060138 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:43.059924    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:43.219191 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:43.219154    4456 patch_prober.go:29] interesting pod/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused" start-of-body=
Nov 02 17:50:43.219412 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:43.219217    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4" podUID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.131:15021/healthz/ready\": dial tcp 10.134.0.131:15021: connect: connection refused"
Nov 02 17:50:43.226990 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:43.226947    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:43.744996 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:43.744959    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:50:43.746187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:43.746160    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:50:44.224549 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.224511    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:50:44.225839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.225799    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:50:44.657741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.657640    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:44.804828 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.804775    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:44.877343 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.877303    4456 scope.go:122] "RemoveContainer" containerID="5781bfe9ba31568fa9879ba01ecac6b559a940287532dad8bfc3e6be3fb46bad"
Nov 02 17:50:44.892411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.892391    4456 scope.go:122] "RemoveContainer" containerID="117f161c65c576cb2dac9940488bef66b32174a7a3539a091f07e77025705b27"
Nov 02 17:50:44.920872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.920843    4456 scope.go:122] "RemoveContainer" containerID="070f0b4104b9e8bce8c871f6cff15ff7db04d4c6c8bba4b100cb1f7808e4f428"
Nov 02 17:50:44.936714 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.936695    4456 scope.go:122] "RemoveContainer" containerID="2a09f157cd7e1080d1b4c88b58c6f1976a542f38d8e86cb18cf7c5a525e335ae"
Nov 02 17:50:44.947356 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.947330    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"]
Nov 02 17:50:44.949884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.949807    4456 scope.go:122] "RemoveContainer" containerID="403652b698825ca949f4d3dab466777f1fa43423b1043ee7926794fe55ea820c"
Nov 02 17:50:44.952208 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.952181    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:44.953185 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.953167    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"
Nov 02 17:50:44.965376 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.965358    4456 scope.go:122] "RemoveContainer" containerID="883eff289f03da617853c7e0d1141e259b3c0a03e1b0569a985e9ed7312a83d2"
Nov 02 17:50:44.979802 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.979775    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"]
Nov 02 17:50:44.980120 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.980096    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-api-b-769b70\""
Nov 02 17:50:44.980298 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.980277    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-api-b-769b70-dockercfg-vb9ql\""
Nov 02 17:50:44.981393 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:44.981370    4456 scope.go:122] "RemoveContainer" containerID="7db6504040396951ad899da09d4506da0ce0fe4b8fbcf7676cefb48c88a256a6"
Nov 02 17:50:45.102842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.095864    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sqtnr\" (UniqueName: \"kubernetes.io/projected/caf4d660-cd9a-4cd9-90bd-4ec5838649ee-kube-api-access-sqtnr\") pod \"payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc\" (UID: \"caf4d660-cd9a-4cd9-90bd-4ec5838649ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"
Nov 02 17:50:45.102842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.095934    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf4d660-cd9a-4cd9-90bd-4ec5838649ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc\" (UID: \"caf4d660-cd9a-4cd9-90bd-4ec5838649ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"
Nov 02 17:50:45.158938 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.158693    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"]
Nov 02 17:50:45.222994 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.221503    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sqtnr\" (UniqueName: \"kubernetes.io/projected/caf4d660-cd9a-4cd9-90bd-4ec5838649ee-kube-api-access-sqtnr\") pod \"payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc\" (UID: \"caf4d660-cd9a-4cd9-90bd-4ec5838649ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"
Nov 02 17:50:45.222994 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.221580    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf4d660-cd9a-4cd9-90bd-4ec5838649ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc\" (UID: \"caf4d660-cd9a-4cd9-90bd-4ec5838649ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"
Nov 02 17:50:45.222994 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.222509    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf4d660-cd9a-4cd9-90bd-4ec5838649ee-plugins-config-volume\") pod \"payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc\" (UID: \"caf4d660-cd9a-4cd9-90bd-4ec5838649ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"
Nov 02 17:50:45.224349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.224325    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"
Nov 02 17:50:45.294255 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.294223    4456 scope.go:122] "RemoveContainer" containerID="9c7da9ce7016b8c9c0c375716d458a01cb215583addc9df9c1edd085becf8cd8"
Nov 02 17:50:45.300483 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.300453    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"]
Nov 02 17:50:45.310770 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.310257    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sqtnr\" (UniqueName: \"kubernetes.io/projected/caf4d660-cd9a-4cd9-90bd-4ec5838649ee-kube-api-access-sqtnr\") pod \"payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc\" (UID: \"caf4d660-cd9a-4cd9-90bd-4ec5838649ee\") " pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"
Nov 02 17:50:45.323600 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.323563    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b00cb5-ecfc-4140-bb2f-2b6cd645d538-plugins-config-volume\") pod \"payload-processing-e2e-api-b-769b70-bf6977767-nwxwj\" (UID: \"d6b00cb5-ecfc-4140-bb2f-2b6cd645d538\") " pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"
Nov 02 17:50:45.323780 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.323758    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sw6ht\" (UniqueName: \"kubernetes.io/projected/d6b00cb5-ecfc-4140-bb2f-2b6cd645d538-kube-api-access-sw6ht\") pod \"payload-processing-e2e-api-b-769b70-bf6977767-nwxwj\" (UID: \"d6b00cb5-ecfc-4140-bb2f-2b6cd645d538\") " pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"
Nov 02 17:50:45.345622 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.345598    4456 scope.go:122] "RemoveContainer" containerID="7866b10813614515a69222809e0dbe89b35284b1b018a1020c172d1e3f131fef"
Nov 02 17:50:45.363493 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.363470    4456 scope.go:122] "RemoveContainer" containerID="0f252d8629ebbd1e827fcd456da140add4ec2e64bce2994766ebd2d7897af9d6"
Nov 02 17:50:45.426106 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.426072    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b00cb5-ecfc-4140-bb2f-2b6cd645d538-plugins-config-volume\") pod \"payload-processing-e2e-api-b-769b70-bf6977767-nwxwj\" (UID: \"d6b00cb5-ecfc-4140-bb2f-2b6cd645d538\") " pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"
Nov 02 17:50:45.426219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.426195    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sw6ht\" (UniqueName: \"kubernetes.io/projected/d6b00cb5-ecfc-4140-bb2f-2b6cd645d538-kube-api-access-sw6ht\") pod \"payload-processing-e2e-api-b-769b70-bf6977767-nwxwj\" (UID: \"d6b00cb5-ecfc-4140-bb2f-2b6cd645d538\") " pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"
Nov 02 17:50:45.427251 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.427225    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b00cb5-ecfc-4140-bb2f-2b6cd645d538-plugins-config-volume\") pod \"payload-processing-e2e-api-b-769b70-bf6977767-nwxwj\" (UID: \"d6b00cb5-ecfc-4140-bb2f-2b6cd645d538\") " pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"
Nov 02 17:50:45.460423 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.460386    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sw6ht\" (UniqueName: \"kubernetes.io/projected/d6b00cb5-ecfc-4140-bb2f-2b6cd645d538-kube-api-access-sw6ht\") pod \"payload-processing-e2e-api-b-769b70-bf6977767-nwxwj\" (UID: \"d6b00cb5-ecfc-4140-bb2f-2b6cd645d538\") " pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"
Nov 02 17:50:45.571596 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.571569    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"
Nov 02 17:50:45.645396 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.645360    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"
Nov 02 17:50:45.893690 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:45.893646    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"]
Nov 02 17:50:45.894112 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:50:45.894090    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcaf4d660_cd9a_4cd9_90bd_4ec5838649ee.slice/crio-259cdef8ec0bb0ba31b0808afe9fae14dc4554c07dbcb6269ce360a21ebd1e84 WatchSource:0}: Error finding container 259cdef8ec0bb0ba31b0808afe9fae14dc4554c07dbcb6269ce360a21ebd1e84: Status 404 returned error can't find the container with id 259cdef8ec0bb0ba31b0808afe9fae14dc4554c07dbcb6269ce360a21ebd1e84
Nov 02 17:50:46.017860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:46.015619    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"]
Nov 02 17:50:46.017860 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:50:46.017691    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd6b00cb5_ecfc_4140_bb2f_2b6cd645d538.slice/crio-72f833f70c48a66e3b942031121c20db81be5708019e9a69d1109b73766be403 WatchSource:0}: Error finding container 72f833f70c48a66e3b942031121c20db81be5708019e9a69d1109b73766be403: Status 404 returned error can't find the container with id 72f833f70c48a66e3b942031121c20db81be5708019e9a69d1109b73766be403
Nov 02 17:50:46.171067 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:50:46.171031    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:50:46.449354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:46.449320    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:46.530837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:46.530777    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj" event={"ID":"d6b00cb5-ecfc-4140-bb2f-2b6cd645d538","Type":"ContainerStarted","Data":"b0d320f2d9eea81b8d6b8e91983823809d6484978da046f73f25f810f3f9be51"}
Nov 02 17:50:46.531033 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:46.530901    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj" event={"ID":"d6b00cb5-ecfc-4140-bb2f-2b6cd645d538","Type":"ContainerStarted","Data":"72f833f70c48a66e3b942031121c20db81be5708019e9a69d1109b73766be403"}
Nov 02 17:50:46.532241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:46.532215    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"
Nov 02 17:50:46.535436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:46.535405    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc" event={"ID":"caf4d660-cd9a-4cd9-90bd-4ec5838649ee","Type":"ContainerStarted","Data":"f750a28626bdbfc0688b5b8e129b48a985473b483835c3aaf487c933e2f50a49"}
Nov 02 17:50:46.535587 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:46.535460    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc" event={"ID":"caf4d660-cd9a-4cd9-90bd-4ec5838649ee","Type":"ContainerStarted","Data":"259cdef8ec0bb0ba31b0808afe9fae14dc4554c07dbcb6269ce360a21ebd1e84"}
Nov 02 17:50:46.535761 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:46.535732    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"
Nov 02 17:50:46.578725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:46.578674    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj" podStartSLOduration=2.578659846 podStartE2EDuration="2.578659846s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:46.575536082 +0000 UTC m=+1263.066599258" watchObservedRunningTime="2026-11-02 17:50:46.578659846 +0000 UTC m=+1263.069723046"
Nov 02 17:50:46.619588 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:46.619526    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc" podStartSLOduration=2.61950446 podStartE2EDuration="2.61950446s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:46.61730665 +0000 UTC m=+1263.108369812" watchObservedRunningTime="2026-11-02 17:50:46.61950446 +0000 UTC m=+1263.110567621"
Nov 02 17:50:47.482439 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:47.482405    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"]
Nov 02 17:50:47.483015 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:47.482690    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g" podUID="8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" containerName="istio-proxy" containerID="cri-o://a62cadeb98b862ad8413a6f6bc577441b9d503e25033aa1deeac9e06e9d18660" gracePeriod=30
Nov 02 17:50:47.544256 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:47.544219    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"]
Nov 02 17:50:47.544776 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:47.544743    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b" podUID="dc7a0bba-95b4-4325-8775-a0a73ff1d67a" containerName="istio-proxy" containerID="cri-o://1f3a978598430e7c27106ecf598ecc76acc2a9ae638e68714d12e4f949cf8947" gracePeriod=30
Nov 02 17:50:48.277942 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:48.277890    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:50.135608 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:50.135574    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:52.072492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.072438    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:52.614403 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.614366    4456 generic.go:356] "Generic (PLEG): container finished" podID="8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" containerID="a62cadeb98b862ad8413a6f6bc577441b9d503e25033aa1deeac9e06e9d18660" exitCode=0
Nov 02 17:50:52.614541 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.614431    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g" event={"ID":"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42","Type":"ContainerDied","Data":"a62cadeb98b862ad8413a6f6bc577441b9d503e25033aa1deeac9e06e9d18660"}
Nov 02 17:50:52.874757 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.874730    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:50:52.897800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.897777    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:50:52.963454 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.963419    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-credential-socket\") pod \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") "
Nov 02 17:50:52.963643 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.963468    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-workload-certs\") pod \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") "
Nov 02 17:50:52.963643 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.963561    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-token\" (UniqueName: \"kubernetes.io/projected/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-token\") pod \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") "
Nov 02 17:50:52.963643 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.963632    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-workload-socket\") pod \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") "
Nov 02 17:50:52.963875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.963739    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-credential-socket" pod "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" (UID: "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:52.963875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.963757    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-envoy\") pod \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") "
Nov 02 17:50:52.964007 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.963879    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-workload-certs" pod "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" (UID: "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:52.964007 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.963941    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-data\") pod \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") "
Nov 02 17:50:52.964007 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.963991    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-podinfo\") pod \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") "
Nov 02 17:50:52.964168 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.964005    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-workload-socket" pod "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" (UID: "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:52.964168 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.964051    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-kube-api-access-798kx\" (UniqueName: \"kubernetes.io/projected/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-kube-api-access-798kx\") pod \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") "
Nov 02 17:50:52.964168 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.964142    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istiod-ca-cert\") pod \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\" (UID: \"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42\") "
Nov 02 17:50:52.964331 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.964175    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-data" pod "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" (UID: "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:52.964729 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.964576    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istiod-ca-cert" pod "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" (UID: "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:52.966039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.965356    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:52.966039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.965384    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:52.966039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.965398    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:52.966039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.965411    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:52.966039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.965425    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:52.966644 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.966620    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-envoy" pod "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" (UID: "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:52.966924 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.966894    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-token" pod "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" (UID: "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:52.967194 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.967173    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-podinfo" pod "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" (UID: "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:50:52.967515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:52.967490    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-kube-api-access-798kx" pod "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" (UID: "8a700a1f-2010-4ca5-a78b-4cae0c8b4d42"). InnerVolumeSpecName "kube-api-access-798kx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:53.066363 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.066271    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-data\") pod \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") "
Nov 02 17:50:53.066517 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.066393    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-credential-socket\") pod \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") "
Nov 02 17:50:53.066517 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.066479    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-podinfo\") pod \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") "
Nov 02 17:50:53.066517 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.066506    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-token\" (UniqueName: \"kubernetes.io/projected/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-token\") pod \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") "
Nov 02 17:50:53.066680 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.066565    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-data" pod "dc7a0bba-95b4-4325-8775-a0a73ff1d67a" (UID: "dc7a0bba-95b4-4325-8775-a0a73ff1d67a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:53.066680 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.066626    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-workload-socket\") pod \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") "
Nov 02 17:50:53.066782 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.066684    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-envoy\") pod \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") "
Nov 02 17:50:53.066782 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.066740    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-credential-socket" pod "dc7a0bba-95b4-4325-8775-a0a73ff1d67a" (UID: "dc7a0bba-95b4-4325-8775-a0a73ff1d67a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:53.066782 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.066750    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-workload-certs\") pod \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") "
Nov 02 17:50:53.066958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.066797    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-kube-api-access-qncn2\" (UniqueName: \"kubernetes.io/projected/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-kube-api-access-qncn2\") pod \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") "
Nov 02 17:50:53.066958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.066891    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istiod-ca-cert\") pod \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\" (UID: \"dc7a0bba-95b4-4325-8775-a0a73ff1d67a\") "
Nov 02 17:50:53.068419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.067486    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-workload-socket" pod "dc7a0bba-95b4-4325-8775-a0a73ff1d67a" (UID: "dc7a0bba-95b4-4325-8775-a0a73ff1d67a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:53.068419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.067864    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-workload-certs" pod "dc7a0bba-95b4-4325-8775-a0a73ff1d67a" (UID: "dc7a0bba-95b4-4325-8775-a0a73ff1d67a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:53.068419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.068011    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:53.068419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.068061    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:53.068419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.068077    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:53.068419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.068094    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:53.068419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.068110    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-798kx\" (UniqueName: \"kubernetes.io/projected/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-kube-api-access-798kx\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:53.068419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.068123    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:53.068419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.068137    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:53.068419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.068151    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:53.068419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.068184    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istiod-ca-cert" pod "dc7a0bba-95b4-4325-8775-a0a73ff1d67a" (UID: "dc7a0bba-95b4-4325-8775-a0a73ff1d67a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:50:53.069576 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.069553    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-podinfo" pod "dc7a0bba-95b4-4325-8775-a0a73ff1d67a" (UID: "dc7a0bba-95b4-4325-8775-a0a73ff1d67a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:50:53.069864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.069807    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-token" pod "dc7a0bba-95b4-4325-8775-a0a73ff1d67a" (UID: "dc7a0bba-95b4-4325-8775-a0a73ff1d67a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:53.069864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.069852    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-envoy" pod "dc7a0bba-95b4-4325-8775-a0a73ff1d67a" (UID: "dc7a0bba-95b4-4325-8775-a0a73ff1d67a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:50:53.070351 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.070333    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-kube-api-access-qncn2" pod "dc7a0bba-95b4-4325-8775-a0a73ff1d67a" (UID: "dc7a0bba-95b4-4325-8775-a0a73ff1d67a"). InnerVolumeSpecName "kube-api-access-qncn2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:50:53.170234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.170201    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:53.170234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.170231    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:53.170234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.170240    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:53.170704 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.170248    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:53.170704 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.170260    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qncn2\" (UniqueName: \"kubernetes.io/projected/dc7a0bba-95b4-4325-8775-a0a73ff1d67a-kube-api-access-qncn2\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:50:53.630845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.630520    4456 generic.go:356] "Generic (PLEG): container finished" podID="dc7a0bba-95b4-4325-8775-a0a73ff1d67a" containerID="1f3a978598430e7c27106ecf598ecc76acc2a9ae638e68714d12e4f949cf8947" exitCode=0
Nov 02 17:50:53.630845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.630592    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"
Nov 02 17:50:53.630845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.630673    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b" event={"ID":"dc7a0bba-95b4-4325-8775-a0a73ff1d67a","Type":"ContainerDied","Data":"1f3a978598430e7c27106ecf598ecc76acc2a9ae638e68714d12e4f949cf8947"}
Nov 02 17:50:53.630845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.630710    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b" event={"ID":"dc7a0bba-95b4-4325-8775-a0a73ff1d67a","Type":"ContainerDied","Data":"f8b30213b2383eb24cd57f5c36f35cf41d38efe10771bd97060fc122460d90da"}
Nov 02 17:50:53.630845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.630731    4456 scope.go:122] "RemoveContainer" containerID="1f3a978598430e7c27106ecf598ecc76acc2a9ae638e68714d12e4f949cf8947"
Nov 02 17:50:53.636263 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.636138    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"
Nov 02 17:50:53.636263 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.636191    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g" event={"ID":"8a700a1f-2010-4ca5-a78b-4cae0c8b4d42","Type":"ContainerDied","Data":"5e1574f81545ba1e233d0f6e36aad2b74f57ac8fb1c4e690469415bdfe862e21"}
Nov 02 17:50:53.647224 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.647201    4456 scope.go:122] "RemoveContainer" containerID="1f3a978598430e7c27106ecf598ecc76acc2a9ae638e68714d12e4f949cf8947"
Nov 02 17:50:53.647531 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:50:53.647511    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3a978598430e7c27106ecf598ecc76acc2a9ae638e68714d12e4f949cf8947\": container with ID starting with 1f3a978598430e7c27106ecf598ecc76acc2a9ae638e68714d12e4f949cf8947 not found: ID does not exist" containerID="1f3a978598430e7c27106ecf598ecc76acc2a9ae638e68714d12e4f949cf8947"
Nov 02 17:50:53.647605 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.647538    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3a978598430e7c27106ecf598ecc76acc2a9ae638e68714d12e4f949cf8947"} err="failed to get container status \"1f3a978598430e7c27106ecf598ecc76acc2a9ae638e68714d12e4f949cf8947\": rpc error: code = NotFound desc = could not find container \"1f3a978598430e7c27106ecf598ecc76acc2a9ae638e68714d12e4f949cf8947\": container with ID starting with 1f3a978598430e7c27106ecf598ecc76acc2a9ae638e68714d12e4f949cf8947 not found: ID does not exist"
Nov 02 17:50:53.647605 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.647555    4456 scope.go:122] "RemoveContainer" containerID="a62cadeb98b862ad8413a6f6bc577441b9d503e25033aa1deeac9e06e9d18660"
Nov 02 17:50:53.710473 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.710441    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"]
Nov 02 17:50:53.717147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.717121    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-fxm5b"]
Nov 02 17:50:53.804102 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.804010    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"]
Nov 02 17:50:53.809321 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.809293    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-h5r7g"]
Nov 02 17:50:53.837165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.836942    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"]
Nov 02 17:50:53.838647 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.838619    4456 cpu_manager.go:414] "containerMap: removing container" podUID="dc7a0bba-95b4-4325-8775-a0a73ff1d67a" containerName="istio-proxy"
Nov 02 17:50:53.838647 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.838645    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dc7a0bba-95b4-4325-8775-a0a73ff1d67a" containerName="istio-proxy"
Nov 02 17:50:53.838888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.838719    4456 cpu_manager.go:414] "containerMap: removing container" podUID="8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" containerName="istio-proxy"
Nov 02 17:50:53.838888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.838728    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" containerName="istio-proxy"
Nov 02 17:50:53.839017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.838964    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" containerName="istio-proxy"
Nov 02 17:50:53.839017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.838989    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="dc7a0bba-95b4-4325-8775-a0a73ff1d67a" containerName="istio-proxy"
Nov 02 17:50:53.844089 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.844067    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:50:53.864247 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.864212    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-2672ed-serving-cert\""
Nov 02 17:50:53.874307 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.874259    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"]
Nov 02 17:50:53.933489 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.933444    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:53.983389 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.983349    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:50:53.983554 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.983519    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:50:53.983691 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.983673    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:50:53.983786 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:53.983775    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hmzvt\" (UniqueName: \"kubernetes.io/projected/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-kube-api-access-hmzvt\") pod \"maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:50:54.089161 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:54.085724    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:50:54.089161 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:54.085905    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hmzvt\" (UniqueName: \"kubernetes.io/projected/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-kube-api-access-hmzvt\") pod \"maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:50:54.089161 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:54.086050    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:50:54.089161 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:54.086175    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:50:54.093841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:54.092363    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-maas-api-tls\") pod \"maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:50:54.093841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:54.093019    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-openshift-service-ca\") pod \"maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:50:54.094843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:54.094490    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:54.101879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:54.100688    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-metrics-certs\") pod \"maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:50:54.127285 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:54.127238    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hmzvt\" (UniqueName: \"kubernetes.io/projected/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-kube-api-access-hmzvt\") pod \"maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") " pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:50:54.184867 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:54.184799    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:50:54.254898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:54.252059    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:54.528391 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:54.528358    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"]
Nov 02 17:50:54.529791 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:50:54.529760    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2d24bd_15b2_4eef_a5fc_5cfc3be9417f.slice/crio-cce0ef7eba3b852d6ef66e46afdd1ea0bcf24431ac65845ef1aa97ca7f6135e2 WatchSource:0}: Error finding container cce0ef7eba3b852d6ef66e46afdd1ea0bcf24431ac65845ef1aa97ca7f6135e2: Status 404 returned error can't find the container with id cce0ef7eba3b852d6ef66e46afdd1ea0bcf24431ac65845ef1aa97ca7f6135e2
Nov 02 17:50:54.531958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:54.531934    4456 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 17:50:54.654638 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:54.654596    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" event={"ID":"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f","Type":"ContainerStarted","Data":"cce0ef7eba3b852d6ef66e46afdd1ea0bcf24431ac65845ef1aa97ca7f6135e2"}
Nov 02 17:50:55.621340 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:55.621304    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a700a1f-2010-4ca5-a78b-4cae0c8b4d42" path="/var/lib/kubelet/pods/8a700a1f-2010-4ca5-a78b-4cae0c8b4d42/volumes"
Nov 02 17:50:55.622153 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:55.622134    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dc7a0bba-95b4-4325-8775-a0a73ff1d67a" path="/var/lib/kubelet/pods/dc7a0bba-95b4-4325-8775-a0a73ff1d67a/volumes"
Nov 02 17:50:55.668183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:55.668149    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" event={"ID":"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f","Type":"ContainerStarted","Data":"3041e8bd23c8eb7470f8dfba16bd766c148c2bcbf3b2edae11533a65a4f804ed"}
Nov 02 17:50:55.668395 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:55.668373    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:50:55.716271 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:55.716213    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podStartSLOduration=2.374438679 podStartE2EDuration="2.716198589s" totalImagesPullingTime="341.75991ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:50:53 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:50:55.711095606 +0000 UTC m=+1272.202158768" watchObservedRunningTime="2026-11-02 17:50:55.716198589 +0000 UTC m=+1272.207261792"
Nov 02 17:50:55.769861 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:55.769828    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:57.555525 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:57.555489    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:50:57.578164 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:57.577805    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"
Nov 02 17:50:58.575573 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:58.575547    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"
Nov 02 17:50:59.470394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:50:59.465383    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:01.288208 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:01.288154    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:01.684644 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:01.684002    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 17:51:02.847839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:02.847311    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:03.317261 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:03.317224    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"]
Nov 02 17:51:03.317560 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:03.317532    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44" podUID="c5f3dfbb-c7df-4c2b-8531-983517c22929" containerName="istio-proxy" containerID="cri-o://88422983dda55c734dfd03b064a332cab84b26aa5c593416425202c7dae4255f" gracePeriod=30
Nov 02 17:51:03.322552 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:03.322523    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"]
Nov 02 17:51:03.322777 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:03.322743    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs" podUID="35675ac4-845c-4700-95c7-9f3ba72bd20a" containerName="istio-proxy" containerID="cri-o://45d088cb196f174e304b13054a3ac9be6e5ffbe0a3e532e8ace5a77828a452ff" gracePeriod=30
Nov 02 17:51:03.413476 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:03.413435    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:03.577215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:03.577124    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:03.979065 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:03.979030    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"]
Nov 02 17:51:04.022195 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.022158    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.033755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.033725    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"]
Nov 02 17:51:04.096836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.095150    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-1e5fcb-openshift-default-dockercfg-7f6dg\""
Nov 02 17:51:04.147149 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.147112    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:04.165841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.153002    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37acee8-ead8-40d3-8d09-160e87271744-istiod-ca-cert\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.191841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.166150    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-istio-data\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.192239 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.192206    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-workload-certs\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.192489 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.192472    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-istio-envoy\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.195837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.192694    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37acee8-ead8-40d3-8d09-160e87271744-istio-podinfo\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.208741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.208706    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-46l26\" (UniqueName: \"kubernetes.io/projected/e37acee8-ead8-40d3-8d09-160e87271744-kube-api-access-46l26\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.208976 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.208959    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-workload-socket\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.209072 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.209057    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-credential-socket\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.209306 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.209293    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37acee8-ead8-40d3-8d09-160e87271744-istio-token\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.315114 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.315026    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:04.316548 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.316518    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37acee8-ead8-40d3-8d09-160e87271744-istio-podinfo\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.317396 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.317370    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-46l26\" (UniqueName: \"kubernetes.io/projected/e37acee8-ead8-40d3-8d09-160e87271744-kube-api-access-46l26\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.317968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.317948    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-workload-socket\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.325311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.322950    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-credential-socket\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.325311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.323115    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37acee8-ead8-40d3-8d09-160e87271744-istio-token\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.325311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.323179    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37acee8-ead8-40d3-8d09-160e87271744-istiod-ca-cert\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.325311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.323245    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-istio-data\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.325311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.323358    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-workload-certs\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.325311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.323483    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-istio-envoy\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.325311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.318402    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-workload-socket\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.325311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.324519    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-credential-socket\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.325311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.324841    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-istio-data\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.325917 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.325492    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37acee8-ead8-40d3-8d09-160e87271744-istiod-ca-cert\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.325917 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.325738    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-workload-certs\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.333310 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.333269    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-istio-envoy\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.342578 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.335420    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37acee8-ead8-40d3-8d09-160e87271744-istio-podinfo\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.352704 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.352672    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-46l26\" (UniqueName: \"kubernetes.io/projected/e37acee8-ead8-40d3-8d09-160e87271744-kube-api-access-46l26\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.357987 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.357957    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37acee8-ead8-40d3-8d09-160e87271744-istio-token\") pod \"e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") " pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:04.642146 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:04.642023    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:05.020721 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:51:05.020177    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode37acee8_ead8_40d3_8d09_160e87271744.slice/crio-4fecc905f5856a6bcd63ec919ff4fd815425823acf0332dabf3a7765f005e8e8 WatchSource:0}: Error finding container 4fecc905f5856a6bcd63ec919ff4fd815425823acf0332dabf3a7765f005e8e8: Status 404 returned error can't find the container with id 4fecc905f5856a6bcd63ec919ff4fd815425823acf0332dabf3a7765f005e8e8
Nov 02 17:51:05.023093 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:05.023066    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"]
Nov 02 17:51:05.023188 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:05.023154    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:05.023226 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:05.023212    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:05.023294 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:05.023241    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:05.408892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:05.396256    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:05.843747 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:05.843706    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq" event={"ID":"e37acee8-ead8-40d3-8d09-160e87271744","Type":"ContainerStarted","Data":"9b2e7216425c06bb756659c307e4b41ecc390921f7e7809ec975eebf56da772c"}
Nov 02 17:51:05.844114 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:05.843770    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq" event={"ID":"e37acee8-ead8-40d3-8d09-160e87271744","Type":"ContainerStarted","Data":"4fecc905f5856a6bcd63ec919ff4fd815425823acf0332dabf3a7765f005e8e8"}
Nov 02 17:51:05.922059 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:05.922005    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq" podStartSLOduration=2.921989449 podStartE2EDuration="2.921989449s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:03 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:05.918835221 +0000 UTC m=+1282.409898379" watchObservedRunningTime="2026-11-02 17:51:05.921989449 +0000 UTC m=+1282.413052609"
Nov 02 17:51:06.643290 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:06.643251    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:06.649615 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:06.649590    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:06.853132 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:06.852914    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:06.853894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:06.853875    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:51:07.207840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:07.207703    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:08.428478 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.428435    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:08.588101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.587987    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:08.688706 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.688372    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:51:08.695078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.695052    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:51:08.864469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.864375    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-workload-socket\") pod \"35675ac4-845c-4700-95c7-9f3ba72bd20a\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") "
Nov 02 17:51:08.864469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.864451    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-envoy\") pod \"35675ac4-845c-4700-95c7-9f3ba72bd20a\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") "
Nov 02 17:51:08.864706 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.864522    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-envoy\") pod \"c5f3dfbb-c7df-4c2b-8531-983517c22929\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") "
Nov 02 17:51:08.864706 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.864558    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-credential-socket\") pod \"c5f3dfbb-c7df-4c2b-8531-983517c22929\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") "
Nov 02 17:51:08.864706 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.864592    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-credential-socket\") pod \"35675ac4-845c-4700-95c7-9f3ba72bd20a\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") "
Nov 02 17:51:08.864706 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.864637    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-podinfo\") pod \"35675ac4-845c-4700-95c7-9f3ba72bd20a\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") "
Nov 02 17:51:08.864912 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.864800    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-workload-certs\") pod \"35675ac4-845c-4700-95c7-9f3ba72bd20a\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") "
Nov 02 17:51:08.864912 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.864850    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-podinfo\") pod \"c5f3dfbb-c7df-4c2b-8531-983517c22929\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") "
Nov 02 17:51:08.864912 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.864889    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-token\" (UniqueName: \"kubernetes.io/projected/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-token\") pod \"35675ac4-845c-4700-95c7-9f3ba72bd20a\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") "
Nov 02 17:51:08.865021 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.864972    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-workload-certs\") pod \"c5f3dfbb-c7df-4c2b-8531-983517c22929\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") "
Nov 02 17:51:08.865021 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.864998    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-token\" (UniqueName: \"kubernetes.io/projected/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-token\") pod \"c5f3dfbb-c7df-4c2b-8531-983517c22929\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") "
Nov 02 17:51:08.865178 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.865162    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-data\") pod \"c5f3dfbb-c7df-4c2b-8531-983517c22929\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") "
Nov 02 17:51:08.865233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.865217    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/35675ac4-845c-4700-95c7-9f3ba72bd20a-kube-api-access-56xc8\" (UniqueName: \"kubernetes.io/projected/35675ac4-845c-4700-95c7-9f3ba72bd20a-kube-api-access-56xc8\") pod \"35675ac4-845c-4700-95c7-9f3ba72bd20a\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") "
Nov 02 17:51:08.865294 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.865280    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-data\") pod \"35675ac4-845c-4700-95c7-9f3ba72bd20a\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") "
Nov 02 17:51:08.865332 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.865317    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/35675ac4-845c-4700-95c7-9f3ba72bd20a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35675ac4-845c-4700-95c7-9f3ba72bd20a-istiod-ca-cert\") pod \"35675ac4-845c-4700-95c7-9f3ba72bd20a\" (UID: \"35675ac4-845c-4700-95c7-9f3ba72bd20a\") "
Nov 02 17:51:08.865395 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.865381    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c5f3dfbb-c7df-4c2b-8531-983517c22929-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f3dfbb-c7df-4c2b-8531-983517c22929-istiod-ca-cert\") pod \"c5f3dfbb-c7df-4c2b-8531-983517c22929\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") "
Nov 02 17:51:08.865429 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.865415    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c5f3dfbb-c7df-4c2b-8531-983517c22929-kube-api-access-px957\" (UniqueName: \"kubernetes.io/projected/c5f3dfbb-c7df-4c2b-8531-983517c22929-kube-api-access-px957\") pod \"c5f3dfbb-c7df-4c2b-8531-983517c22929\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") "
Nov 02 17:51:08.865470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.865462    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-workload-socket\") pod \"c5f3dfbb-c7df-4c2b-8531-983517c22929\" (UID: \"c5f3dfbb-c7df-4c2b-8531-983517c22929\") "
Nov 02 17:51:08.866660 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.866634    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-workload-socket" pod "c5f3dfbb-c7df-4c2b-8531-983517c22929" (UID: "c5f3dfbb-c7df-4c2b-8531-983517c22929"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:08.866873 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.866855    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-workload-socket" pod "35675ac4-845c-4700-95c7-9f3ba72bd20a" (UID: "35675ac4-845c-4700-95c7-9f3ba72bd20a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:08.869048 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.869021    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-envoy" pod "35675ac4-845c-4700-95c7-9f3ba72bd20a" (UID: "35675ac4-845c-4700-95c7-9f3ba72bd20a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:08.869579 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.869546    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-workload-certs" pod "c5f3dfbb-c7df-4c2b-8531-983517c22929" (UID: "c5f3dfbb-c7df-4c2b-8531-983517c22929"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:08.869767 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.869747    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-credential-socket" pod "c5f3dfbb-c7df-4c2b-8531-983517c22929" (UID: "c5f3dfbb-c7df-4c2b-8531-983517c22929"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:08.869999 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.869979    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-credential-socket" pod "35675ac4-845c-4700-95c7-9f3ba72bd20a" (UID: "35675ac4-845c-4700-95c7-9f3ba72bd20a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:08.870614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.870551    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-workload-certs" pod "35675ac4-845c-4700-95c7-9f3ba72bd20a" (UID: "35675ac4-845c-4700-95c7-9f3ba72bd20a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:08.870614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.870556    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-data" pod "c5f3dfbb-c7df-4c2b-8531-983517c22929" (UID: "c5f3dfbb-c7df-4c2b-8531-983517c22929"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:08.870943 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.870884    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c5f3dfbb-c7df-4c2b-8531-983517c22929-istiod-ca-cert" pod "c5f3dfbb-c7df-4c2b-8531-983517c22929" (UID: "c5f3dfbb-c7df-4c2b-8531-983517c22929"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:08.871085 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.871064    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-data" pod "35675ac4-845c-4700-95c7-9f3ba72bd20a" (UID: "35675ac4-845c-4700-95c7-9f3ba72bd20a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:08.871172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.871140    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/35675ac4-845c-4700-95c7-9f3ba72bd20a-istiod-ca-cert" pod "35675ac4-845c-4700-95c7-9f3ba72bd20a" (UID: "35675ac4-845c-4700-95c7-9f3ba72bd20a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:08.875299 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.875277    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-envoy" pod "c5f3dfbb-c7df-4c2b-8531-983517c22929" (UID: "c5f3dfbb-c7df-4c2b-8531-983517c22929"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:51:08.880612 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.880576    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f3dfbb-c7df-4c2b-8531-983517c22929-kube-api-access-px957" pod "c5f3dfbb-c7df-4c2b-8531-983517c22929" (UID: "c5f3dfbb-c7df-4c2b-8531-983517c22929"). InnerVolumeSpecName "kube-api-access-px957". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:08.883101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.882443    4456 generic.go:356] "Generic (PLEG): container finished" podID="c5f3dfbb-c7df-4c2b-8531-983517c22929" containerID="88422983dda55c734dfd03b064a332cab84b26aa5c593416425202c7dae4255f" exitCode=0
Nov 02 17:51:08.883101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.882518    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"
Nov 02 17:51:08.883101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.882566    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44" event={"ID":"c5f3dfbb-c7df-4c2b-8531-983517c22929","Type":"ContainerDied","Data":"88422983dda55c734dfd03b064a332cab84b26aa5c593416425202c7dae4255f"}
Nov 02 17:51:08.883101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.882643    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44" event={"ID":"c5f3dfbb-c7df-4c2b-8531-983517c22929","Type":"ContainerDied","Data":"abbab27e711230b0e1ae5e7c024f18bc32de7047ce2c61096ff9211a8ee86334"}
Nov 02 17:51:08.883101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.882674    4456 scope.go:122] "RemoveContainer" containerID="88422983dda55c734dfd03b064a332cab84b26aa5c593416425202c7dae4255f"
Nov 02 17:51:08.886886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.886257    4456 generic.go:356] "Generic (PLEG): container finished" podID="35675ac4-845c-4700-95c7-9f3ba72bd20a" containerID="45d088cb196f174e304b13054a3ac9be6e5ffbe0a3e532e8ace5a77828a452ff" exitCode=0
Nov 02 17:51:08.887916 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.887589    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"
Nov 02 17:51:08.887916 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.887745    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs" event={"ID":"35675ac4-845c-4700-95c7-9f3ba72bd20a","Type":"ContainerDied","Data":"45d088cb196f174e304b13054a3ac9be6e5ffbe0a3e532e8ace5a77828a452ff"}
Nov 02 17:51:08.887916 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.887783    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs" event={"ID":"35675ac4-845c-4700-95c7-9f3ba72bd20a","Type":"ContainerDied","Data":"6852a35a5d2c5faa28d093dff795ed3b7e2296bbdd025379045f5e200449179c"}
Nov 02 17:51:08.895708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.895314    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-token" pod "35675ac4-845c-4700-95c7-9f3ba72bd20a" (UID: "35675ac4-845c-4700-95c7-9f3ba72bd20a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:08.895708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.895403    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-podinfo" pod "c5f3dfbb-c7df-4c2b-8531-983517c22929" (UID: "c5f3dfbb-c7df-4c2b-8531-983517c22929"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:51:08.895708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.895470    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-token" pod "c5f3dfbb-c7df-4c2b-8531-983517c22929" (UID: "c5f3dfbb-c7df-4c2b-8531-983517c22929"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:08.895708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.895500    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-podinfo" pod "35675ac4-845c-4700-95c7-9f3ba72bd20a" (UID: "35675ac4-845c-4700-95c7-9f3ba72bd20a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:51:08.896746 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.896724    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/35675ac4-845c-4700-95c7-9f3ba72bd20a-kube-api-access-56xc8" pod "35675ac4-845c-4700-95c7-9f3ba72bd20a" (UID: "35675ac4-845c-4700-95c7-9f3ba72bd20a"). InnerVolumeSpecName "kube-api-access-56xc8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:08.901361 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.901342    4456 scope.go:122] "RemoveContainer" containerID="88422983dda55c734dfd03b064a332cab84b26aa5c593416425202c7dae4255f"
Nov 02 17:51:08.901708 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:51:08.901688    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"88422983dda55c734dfd03b064a332cab84b26aa5c593416425202c7dae4255f\": container with ID starting with 88422983dda55c734dfd03b064a332cab84b26aa5c593416425202c7dae4255f not found: ID does not exist" containerID="88422983dda55c734dfd03b064a332cab84b26aa5c593416425202c7dae4255f"
Nov 02 17:51:08.901797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.901716    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"88422983dda55c734dfd03b064a332cab84b26aa5c593416425202c7dae4255f"} err="failed to get container status \"88422983dda55c734dfd03b064a332cab84b26aa5c593416425202c7dae4255f\": rpc error: code = NotFound desc = could not find container \"88422983dda55c734dfd03b064a332cab84b26aa5c593416425202c7dae4255f\": container with ID starting with 88422983dda55c734dfd03b064a332cab84b26aa5c593416425202c7dae4255f not found: ID does not exist"
Nov 02 17:51:08.901797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.901734    4456 scope.go:122] "RemoveContainer" containerID="45d088cb196f174e304b13054a3ac9be6e5ffbe0a3e532e8ace5a77828a452ff"
Nov 02 17:51:08.915272 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.915247    4456 scope.go:122] "RemoveContainer" containerID="45d088cb196f174e304b13054a3ac9be6e5ffbe0a3e532e8ace5a77828a452ff"
Nov 02 17:51:08.917089 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:51:08.915613    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"45d088cb196f174e304b13054a3ac9be6e5ffbe0a3e532e8ace5a77828a452ff\": container with ID starting with 45d088cb196f174e304b13054a3ac9be6e5ffbe0a3e532e8ace5a77828a452ff not found: ID does not exist" containerID="45d088cb196f174e304b13054a3ac9be6e5ffbe0a3e532e8ace5a77828a452ff"
Nov 02 17:51:08.917089 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.915648    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"45d088cb196f174e304b13054a3ac9be6e5ffbe0a3e532e8ace5a77828a452ff"} err="failed to get container status \"45d088cb196f174e304b13054a3ac9be6e5ffbe0a3e532e8ace5a77828a452ff\": rpc error: code = NotFound desc = could not find container \"45d088cb196f174e304b13054a3ac9be6e5ffbe0a3e532e8ace5a77828a452ff\": container with ID starting with 45d088cb196f174e304b13054a3ac9be6e5ffbe0a3e532e8ace5a77828a452ff not found: ID does not exist"
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968055    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968088    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968105    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968118    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968134    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968150    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968164    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968182    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968197    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968209    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968222    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968233    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968245    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-56xc8\" (UniqueName: \"kubernetes.io/projected/35675ac4-845c-4700-95c7-9f3ba72bd20a-kube-api-access-56xc8\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968254    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/35675ac4-845c-4700-95c7-9f3ba72bd20a-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968262    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/35675ac4-845c-4700-95c7-9f3ba72bd20a-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968269    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/c5f3dfbb-c7df-4c2b-8531-983517c22929-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968277    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-px957\" (UniqueName: \"kubernetes.io/projected/c5f3dfbb-c7df-4c2b-8531-983517c22929-kube-api-access-px957\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:08.968837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:08.968284    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/c5f3dfbb-c7df-4c2b-8531-983517c22929-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:09.235683 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:09.235644    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:09.299699 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:09.299668    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"]
Nov 02 17:51:09.305883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:09.305850    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-ftm44"]
Nov 02 17:51:09.325752 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:09.325714    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"]
Nov 02 17:51:09.332631 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:09.332603    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-2j2hs"]
Nov 02 17:51:09.620754 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:09.620674    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="35675ac4-845c-4700-95c7-9f3ba72bd20a" path="/var/lib/kubelet/pods/35675ac4-845c-4700-95c7-9f3ba72bd20a/volumes"
Nov 02 17:51:09.621474 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:09.621453    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c5f3dfbb-c7df-4c2b-8531-983517c22929" path="/var/lib/kubelet/pods/c5f3dfbb-c7df-4c2b-8531-983517c22929/volumes"
Nov 02 17:51:10.113833 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:10.113772    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:10.266525 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:10.266486    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:11.073010 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:11.072972    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:12.855072 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:12.855044    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:12.936835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:12.936782    4456 generic.go:356] "Generic (PLEG): container finished" podID="33530571-d61c-488e-be5b-54e298cfd4e4" containerID="64e1c83b77cad53fbb7765a03a1aae028227679f40fd0cd1509bef2eb59cd3d5" exitCode=0
Nov 02 17:51:12.937067 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:12.936889    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd" event={"ID":"33530571-d61c-488e-be5b-54e298cfd4e4","Type":"ContainerDied","Data":"64e1c83b77cad53fbb7765a03a1aae028227679f40fd0cd1509bef2eb59cd3d5"}
Nov 02 17:51:12.937486 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:12.937465    4456 scope.go:122] "RemoveContainer" containerID="64e1c83b77cad53fbb7765a03a1aae028227679f40fd0cd1509bef2eb59cd3d5"
Nov 02 17:51:12.940232 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:12.940193    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:12.940353 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:12.940291    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:12.940353 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:12.940339    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:13.947721 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:13.947684    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd" event={"ID":"33530571-d61c-488e-be5b-54e298cfd4e4","Type":"ContainerStarted","Data":"85b38081a920ac04873b6d419f556f5ffffef86b5115b351b3b2b35a26058e83"}
Nov 02 17:51:14.569522 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:14.569488    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:51:14.570431 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:14.570402    4456 patch_prober.go:29] interesting pod/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 17:51:14.570567 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:14.570460    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd" podUID="33530571-d61c-488e-be5b-54e298cfd4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 17:51:14.656309 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:14.656278    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:51:14.659675 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:14.659647    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:15.433738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.433700    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"]
Nov 02 17:51:15.434693 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.434668    4456 cpu_manager.go:414] "containerMap: removing container" podUID="35675ac4-845c-4700-95c7-9f3ba72bd20a" containerName="istio-proxy"
Nov 02 17:51:15.434693 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.434693    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="35675ac4-845c-4700-95c7-9f3ba72bd20a" containerName="istio-proxy"
Nov 02 17:51:15.434919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.434737    4456 cpu_manager.go:414] "containerMap: removing container" podUID="c5f3dfbb-c7df-4c2b-8531-983517c22929" containerName="istio-proxy"
Nov 02 17:51:15.434919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.434745    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c5f3dfbb-c7df-4c2b-8531-983517c22929" containerName="istio-proxy"
Nov 02 17:51:15.435024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.434985    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="c5f3dfbb-c7df-4c2b-8531-983517c22929" containerName="istio-proxy"
Nov 02 17:51:15.435024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.435015    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="35675ac4-845c-4700-95c7-9f3ba72bd20a" containerName="istio-proxy"
Nov 02 17:51:15.437881 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.437863    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"
Nov 02 17:51:15.457047 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.457025    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"]
Nov 02 17:51:15.467241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.463873    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:51:15.467241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.463891    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:51:15.570006 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.569976    4456 patch_prober.go:29] interesting pod/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 17:51:15.570188 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.570035    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd" podUID="33530571-d61c-488e-be5b-54e298cfd4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 17:51:15.576987 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.576955    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0bbc0b5-0c76-4471-950b-b07db0a89542-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-82fwt\" (UID: \"c0bbc0b5-0c76-4471-950b-b07db0a89542\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"
Nov 02 17:51:15.577154 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.577067    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j6skk\" (UniqueName: \"kubernetes.io/projected/c0bbc0b5-0c76-4471-950b-b07db0a89542-kube-api-access-j6skk\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-82fwt\" (UID: \"c0bbc0b5-0c76-4471-950b-b07db0a89542\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"
Nov 02 17:51:15.577316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.577302    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0bbc0b5-0c76-4471-950b-b07db0a89542-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-82fwt\" (UID: \"c0bbc0b5-0c76-4471-950b-b07db0a89542\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"
Nov 02 17:51:15.678501 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.678461    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0bbc0b5-0c76-4471-950b-b07db0a89542-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-82fwt\" (UID: \"c0bbc0b5-0c76-4471-950b-b07db0a89542\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"
Nov 02 17:51:15.678719 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.678565    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0bbc0b5-0c76-4471-950b-b07db0a89542-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-82fwt\" (UID: \"c0bbc0b5-0c76-4471-950b-b07db0a89542\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"
Nov 02 17:51:15.678719 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.678633    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j6skk\" (UniqueName: \"kubernetes.io/projected/c0bbc0b5-0c76-4471-950b-b07db0a89542-kube-api-access-j6skk\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-82fwt\" (UID: \"c0bbc0b5-0c76-4471-950b-b07db0a89542\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"
Nov 02 17:51:15.679222 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.679200    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0bbc0b5-0c76-4471-950b-b07db0a89542-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-82fwt\" (UID: \"c0bbc0b5-0c76-4471-950b-b07db0a89542\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"
Nov 02 17:51:15.713313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.713241    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0bbc0b5-0c76-4471-950b-b07db0a89542-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-82fwt\" (UID: \"c0bbc0b5-0c76-4471-950b-b07db0a89542\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"
Nov 02 17:51:15.717069 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.717043    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j6skk\" (UniqueName: \"kubernetes.io/projected/c0bbc0b5-0c76-4471-950b-b07db0a89542-kube-api-access-j6skk\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-82fwt\" (UID: \"c0bbc0b5-0c76-4471-950b-b07db0a89542\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"
Nov 02 17:51:15.751638 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:15.751611    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"
Nov 02 17:51:16.117864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:16.117807    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"]
Nov 02 17:51:16.168879 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:51:16.168845    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:51:16.570722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:16.570691    4456 patch_prober.go:29] interesting pod/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 17:51:16.571236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:16.570746    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd" podUID="33530571-d61c-488e-be5b-54e298cfd4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 17:51:16.579707 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:16.579461    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:16.988180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:16.988144    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt" event={"ID":"c0bbc0b5-0c76-4471-950b-b07db0a89542","Type":"ContainerStarted","Data":"b7b1ff2346b373c8a5ff7b17629bb45aecfe81aa4915e82fb8d4cb26884b87a7"}
Nov 02 17:51:16.988535 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:16.988512    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt" event={"ID":"c0bbc0b5-0c76-4471-950b-b07db0a89542","Type":"ContainerStarted","Data":"913e9d7d5ee1d466f5502c05c10c9d265b58d7883be736e44fe8b169fd24da3d"}
Nov 02 17:51:17.041715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:17.041646    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt" podStartSLOduration=2.041625717 podStartE2EDuration="2.041625717s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:17.039378798 +0000 UTC m=+1293.530441984" watchObservedRunningTime="2026-11-02 17:51:17.041625717 +0000 UTC m=+1293.532688879"
Nov 02 17:51:17.570200 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:17.570166    4456 patch_prober.go:29] interesting pod/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 17:51:17.570376 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:17.570236    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd" podUID="33530571-d61c-488e-be5b-54e298cfd4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 17:51:18.011296 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:18.011257    4456 generic.go:356] "Generic (PLEG): container finished" podID="c0bbc0b5-0c76-4471-950b-b07db0a89542" containerID="b7b1ff2346b373c8a5ff7b17629bb45aecfe81aa4915e82fb8d4cb26884b87a7" exitCode=0
Nov 02 17:51:18.011707 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:18.011336    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt" event={"ID":"c0bbc0b5-0c76-4471-950b-b07db0a89542","Type":"ContainerDied","Data":"b7b1ff2346b373c8a5ff7b17629bb45aecfe81aa4915e82fb8d4cb26884b87a7"}
Nov 02 17:51:18.403902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:18.379944    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:18.570285 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:18.570247    4456 patch_prober.go:29] interesting pod/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused" start-of-body=
Nov 02 17:51:18.570468 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:18.570307    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd" podUID="33530571-d61c-488e-be5b-54e298cfd4e4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.140:15021/healthz/ready\": dial tcp 10.134.0.140:15021: connect: connection refused"
Nov 02 17:51:19.227930 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:19.227897    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"
Nov 02 17:51:19.354574 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:19.352534    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/c0bbc0b5-0c76-4471-950b-b07db0a89542-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0bbc0b5-0c76-4471-950b-b07db0a89542-maas-api-xks-ca\") pod \"c0bbc0b5-0c76-4471-950b-b07db0a89542\" (UID: \"c0bbc0b5-0c76-4471-950b-b07db0a89542\") "
Nov 02 17:51:19.354574 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:19.352733    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0bbc0b5-0c76-4471-950b-b07db0a89542-kube-api-access-j6skk\" (UniqueName: \"kubernetes.io/projected/c0bbc0b5-0c76-4471-950b-b07db0a89542-kube-api-access-j6skk\") pod \"c0bbc0b5-0c76-4471-950b-b07db0a89542\" (UID: \"c0bbc0b5-0c76-4471-950b-b07db0a89542\") "
Nov 02 17:51:19.354574 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:19.352978    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0bbc0b5-0c76-4471-950b-b07db0a89542-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0bbc0b5-0c76-4471-950b-b07db0a89542-maas-api-openshift-ca\") pod \"c0bbc0b5-0c76-4471-950b-b07db0a89542\" (UID: \"c0bbc0b5-0c76-4471-950b-b07db0a89542\") "
Nov 02 17:51:19.354907 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:19.354722    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0bbc0b5-0c76-4471-950b-b07db0a89542-maas-api-openshift-ca" pod "c0bbc0b5-0c76-4471-950b-b07db0a89542" (UID: "c0bbc0b5-0c76-4471-950b-b07db0a89542"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:19.362466 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:19.362418    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/c0bbc0b5-0c76-4471-950b-b07db0a89542-maas-api-xks-ca" pod "c0bbc0b5-0c76-4471-950b-b07db0a89542" (UID: "c0bbc0b5-0c76-4471-950b-b07db0a89542"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:51:19.365494 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:19.365444    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0bbc0b5-0c76-4471-950b-b07db0a89542-kube-api-access-j6skk" pod "c0bbc0b5-0c76-4471-950b-b07db0a89542" (UID: "c0bbc0b5-0c76-4471-950b-b07db0a89542"). InnerVolumeSpecName "kube-api-access-j6skk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:19.455939 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:19.455909    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j6skk\" (UniqueName: \"kubernetes.io/projected/c0bbc0b5-0c76-4471-950b-b07db0a89542-kube-api-access-j6skk\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:19.455939 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:19.455937    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/c0bbc0b5-0c76-4471-950b-b07db0a89542-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:19.455939 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:19.455946    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/c0bbc0b5-0c76-4471-950b-b07db0a89542-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:19.575039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:19.575007    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:51:19.576145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:19.576122    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:51:20.030138 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.030106    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"
Nov 02 17:51:20.030354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.030149    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt" event={"ID":"c0bbc0b5-0c76-4471-950b-b07db0a89542","Type":"ContainerDied","Data":"913e9d7d5ee1d466f5502c05c10c9d265b58d7883be736e44fe8b169fd24da3d"}
Nov 02 17:51:20.030354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.030208    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="913e9d7d5ee1d466f5502c05c10c9d265b58d7883be736e44fe8b169fd24da3d"
Nov 02 17:51:20.096990 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.096953    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"]
Nov 02 17:51:20.097749 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.097735    4456 cpu_manager.go:414] "containerMap: removing container" podUID="c0bbc0b5-0c76-4471-950b-b07db0a89542" containerName="revoke-keys"
Nov 02 17:51:20.097795 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.097752    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0bbc0b5-0c76-4471-950b-b07db0a89542" containerName="revoke-keys"
Nov 02 17:51:20.097914 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.097903    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0bbc0b5-0c76-4471-950b-b07db0a89542" containerName="revoke-keys"
Nov 02 17:51:20.100666 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.100648    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:20.134908 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.129945    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-b-769b70-serving-cert\""
Nov 02 17:51:20.138460 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.138413    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"]
Nov 02 17:51:20.258636 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.258600    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:20.276696 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.276652    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602b86d5-0981-4f7e-8b5d-2c364e55e60e-openshift-service-ca\") pod \"maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:20.276899 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.276753    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hwvrt\" (UniqueName: \"kubernetes.io/projected/602b86d5-0981-4f7e-8b5d-2c364e55e60e-kube-api-access-hwvrt\") pod \"maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:20.277117 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.277082    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/602b86d5-0981-4f7e-8b5d-2c364e55e60e-metrics-certs\") pod \"maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:20.277238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.277122    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602b86d5-0981-4f7e-8b5d-2c364e55e60e-maas-api-tls\") pod \"maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:20.379222 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.379120    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hwvrt\" (UniqueName: \"kubernetes.io/projected/602b86d5-0981-4f7e-8b5d-2c364e55e60e-kube-api-access-hwvrt\") pod \"maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:20.379425 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.379406    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/602b86d5-0981-4f7e-8b5d-2c364e55e60e-metrics-certs\") pod \"maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:20.379501 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.379450    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602b86d5-0981-4f7e-8b5d-2c364e55e60e-maas-api-tls\") pod \"maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:20.379609 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.379592    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602b86d5-0981-4f7e-8b5d-2c364e55e60e-openshift-service-ca\") pod \"maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:20.380448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.380416    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602b86d5-0981-4f7e-8b5d-2c364e55e60e-openshift-service-ca\") pod \"maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:20.383250 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.383227    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602b86d5-0981-4f7e-8b5d-2c364e55e60e-maas-api-tls\") pod \"maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:20.387350 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.387324    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/602b86d5-0981-4f7e-8b5d-2c364e55e60e-metrics-certs\") pod \"maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:20.404354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.404273    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hwvrt\" (UniqueName: \"kubernetes.io/projected/602b86d5-0981-4f7e-8b5d-2c364e55e60e-kube-api-access-hwvrt\") pod \"maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:20.416345 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.416314    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:20.420275 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.420241    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:20.580642 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.580604    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:20.743357 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:20.743327    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"]
Nov 02 17:51:20.743485 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:51:20.743432    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod602b86d5_0981_4f7e_8b5d_2c364e55e60e.slice/crio-d0c052d2b5de9330922010bac93ae6beb998c6da53cc144128597d96023d29db WatchSource:0}: Error finding container d0c052d2b5de9330922010bac93ae6beb998c6da53cc144128597d96023d29db: Status 404 returned error can't find the container with id d0c052d2b5de9330922010bac93ae6beb998c6da53cc144128597d96023d29db
Nov 02 17:51:21.042212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:21.042131    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb" event={"ID":"602b86d5-0981-4f7e-8b5d-2c364e55e60e","Type":"ContainerStarted","Data":"d0c052d2b5de9330922010bac93ae6beb998c6da53cc144128597d96023d29db"}
Nov 02 17:51:22.053234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:22.053195    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb" event={"ID":"602b86d5-0981-4f7e-8b5d-2c364e55e60e","Type":"ContainerStarted","Data":"0d89b1a40dda5d882a2077d0325e0db3911431a68ff9fafdd8fcb91ac3a83303"}
Nov 02 17:51:22.053647 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:22.053267    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:22.111458 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:22.111389    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb" podStartSLOduration=1.773768746 podStartE2EDuration="2.111368622s" totalImagesPullingTime="337.599876ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:20 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:22.108595669 +0000 UTC m=+1298.599658830" watchObservedRunningTime="2026-11-02 17:51:22.111368622 +0000 UTC m=+1298.602431785"
Nov 02 17:51:22.166743 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:22.166710    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:24.125859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:24.123620    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:25.807887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:25.807794    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:25.955333 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:25.955297    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"]
Nov 02 17:51:26.091416 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:26.091332    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:26.240195 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:26.240159    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:27.527431 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:27.527390    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:27.675178 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:27.675136    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:28.015080 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:28.015024    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:28.146876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:28.139292    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:51:28.603876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:28.603839    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:29.424509 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:29.424474    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:30.956011 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:30.955967    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb" podUID="839ebaa1-25e5-4db7-abdc-7f55f837bff7" containerName="payload-pre-processing" containerID="cri-o://648f137883d0ad44adb7ec5c721265dd4b4d1dda41c2a3489d6187de719dab72" gracePeriod=25
Nov 02 17:51:31.169409 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:31.169372    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:32.612883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:32.612831    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:32.845361 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:32.845324    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:33.232079 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:33.232027    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:34.321397 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:34.321343    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:34.478276 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:34.478232    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:34.807926 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:34.807892    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"]
Nov 02 17:51:34.813864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:34.813807    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"
Nov 02 17:51:34.822473 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:34.822443    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"]
Nov 02 17:51:34.824771 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:34.824746    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pzwd9\" (UniqueName: \"kubernetes.io/projected/38cc3415-dd2f-4e7a-91a9-f3740f2cc826-kube-api-access-pzwd9\") pod \"payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n\" (UID: \"38cc3415-dd2f-4e7a-91a9-f3740f2cc826\") " pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"
Nov 02 17:51:34.825899 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:34.825872    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38cc3415-dd2f-4e7a-91a9-f3740f2cc826-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n\" (UID: \"38cc3415-dd2f-4e7a-91a9-f3740f2cc826\") " pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"
Nov 02 17:51:34.846926 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:34.846896    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-063f48-dockercfg-m7xwx\""
Nov 02 17:51:34.882297 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:34.882260    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-063f48\""
Nov 02 17:51:34.928422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:34.928388    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pzwd9\" (UniqueName: \"kubernetes.io/projected/38cc3415-dd2f-4e7a-91a9-f3740f2cc826-kube-api-access-pzwd9\") pod \"payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n\" (UID: \"38cc3415-dd2f-4e7a-91a9-f3740f2cc826\") " pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"
Nov 02 17:51:34.928641 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:34.928437    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38cc3415-dd2f-4e7a-91a9-f3740f2cc826-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n\" (UID: \"38cc3415-dd2f-4e7a-91a9-f3740f2cc826\") " pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"
Nov 02 17:51:34.929356 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:34.929334    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38cc3415-dd2f-4e7a-91a9-f3740f2cc826-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n\" (UID: \"38cc3415-dd2f-4e7a-91a9-f3740f2cc826\") " pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"
Nov 02 17:51:34.952375 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:34.952345    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pzwd9\" (UniqueName: \"kubernetes.io/projected/38cc3415-dd2f-4e7a-91a9-f3740f2cc826-kube-api-access-pzwd9\") pod \"payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n\" (UID: \"38cc3415-dd2f-4e7a-91a9-f3740f2cc826\") " pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"
Nov 02 17:51:35.005124 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:35.005081    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:35.161737 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:35.161601    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"
Nov 02 17:51:35.395839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:35.395323    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:35.593721 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:35.593345    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"]
Nov 02 17:51:35.597617 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:51:35.597587    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38cc3415_dd2f_4e7a_91a9_f3740f2cc826.slice/crio-9200dbe1a3f049104ff025fbbb6a25c2154a88fbf44439d968e06ec2cae8072a WatchSource:0}: Error finding container 9200dbe1a3f049104ff025fbbb6a25c2154a88fbf44439d968e06ec2cae8072a: Status 404 returned error can't find the container with id 9200dbe1a3f049104ff025fbbb6a25c2154a88fbf44439d968e06ec2cae8072a
Nov 02 17:51:36.234360 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:36.234323    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n" event={"ID":"38cc3415-dd2f-4e7a-91a9-f3740f2cc826","Type":"ContainerStarted","Data":"3ca3b8e733f06e462266217f185ef0da9df2ed08e368a7c11910091f122469d5"}
Nov 02 17:51:36.234524 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:36.234381    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n" event={"ID":"38cc3415-dd2f-4e7a-91a9-f3740f2cc826","Type":"ContainerStarted","Data":"9200dbe1a3f049104ff025fbbb6a25c2154a88fbf44439d968e06ec2cae8072a"}
Nov 02 17:51:36.234524 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:36.234424    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"
Nov 02 17:51:36.286503 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:36.286446    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n" podStartSLOduration=2.286433958 podStartE2EDuration="2.286433958s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:36.28588618 +0000 UTC m=+1312.776949350" watchObservedRunningTime="2026-11-02 17:51:36.286433958 +0000 UTC m=+1312.777497147"
Nov 02 17:51:37.325997 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:37.325695    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:37.357748 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:37.357712    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb" podUID="839ebaa1-25e5-4db7-abdc-7f55f837bff7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 17:51:38.733040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:38.733009    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:39.088671 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:39.088587    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:40.254358 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:40.254320    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:40.522479 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:40.522399    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:41.159294 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.159260    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:41.475476 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.475445    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"]
Nov 02 17:51:41.487462 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.487433    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"
Nov 02 17:51:41.520392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.520361    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:51:41.520745 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.520725    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:51:41.553295 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.553254    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"]
Nov 02 17:51:41.606433 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.606399    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05136623-c975-421f-83a8-08c9b8f574e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6\" (UID: \"05136623-c975-421f-83a8-08c9b8f574e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"
Nov 02 17:51:41.606646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.606550    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05136623-c975-421f-83a8-08c9b8f574e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6\" (UID: \"05136623-c975-421f-83a8-08c9b8f574e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"
Nov 02 17:51:41.606723 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.606647    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2lm8f\" (UniqueName: \"kubernetes.io/projected/05136623-c975-421f-83a8-08c9b8f574e3-kube-api-access-2lm8f\") pod \"maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6\" (UID: \"05136623-c975-421f-83a8-08c9b8f574e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"
Nov 02 17:51:41.761940 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.761511    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05136623-c975-421f-83a8-08c9b8f574e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6\" (UID: \"05136623-c975-421f-83a8-08c9b8f574e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"
Nov 02 17:51:41.761940 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.761664    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05136623-c975-421f-83a8-08c9b8f574e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6\" (UID: \"05136623-c975-421f-83a8-08c9b8f574e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"
Nov 02 17:51:41.761940 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.761775    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2lm8f\" (UniqueName: \"kubernetes.io/projected/05136623-c975-421f-83a8-08c9b8f574e3-kube-api-access-2lm8f\") pod \"maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6\" (UID: \"05136623-c975-421f-83a8-08c9b8f574e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"
Nov 02 17:51:41.785910 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.763984    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05136623-c975-421f-83a8-08c9b8f574e3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6\" (UID: \"05136623-c975-421f-83a8-08c9b8f574e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"
Nov 02 17:51:41.794638 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.794577    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2lm8f\" (UniqueName: \"kubernetes.io/projected/05136623-c975-421f-83a8-08c9b8f574e3-kube-api-access-2lm8f\") pod \"maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6\" (UID: \"05136623-c975-421f-83a8-08c9b8f574e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"
Nov 02 17:51:41.813666 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.813639    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05136623-c975-421f-83a8-08c9b8f574e3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6\" (UID: \"05136623-c975-421f-83a8-08c9b8f574e3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"
Nov 02 17:51:41.854782 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:41.854730    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"
Nov 02 17:51:42.308910 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:42.308872    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"]
Nov 02 17:51:42.309442 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:51:42.309402    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod05136623_c975_421f_83a8_08c9b8f574e3.slice/crio-3fb29eed3679aaa7193f5ed33f6fab1fe82d86f6e6dd9566b492c47c26c9ebc0 WatchSource:0}: Error finding container 3fb29eed3679aaa7193f5ed33f6fab1fe82d86f6e6dd9566b492c47c26c9ebc0: Status 404 returned error can't find the container with id 3fb29eed3679aaa7193f5ed33f6fab1fe82d86f6e6dd9566b492c47c26c9ebc0
Nov 02 17:51:42.325826 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:42.325785    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6" event={"ID":"05136623-c975-421f-83a8-08c9b8f574e3","Type":"ContainerStarted","Data":"3fb29eed3679aaa7193f5ed33f6fab1fe82d86f6e6dd9566b492c47c26c9ebc0"}
Nov 02 17:51:43.317929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:43.310343    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:43.349700 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:43.348307    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6" event={"ID":"05136623-c975-421f-83a8-08c9b8f574e3","Type":"ContainerStarted","Data":"d5a7f9bff5a598255785a777f7ede52ce5db84493e91ffbd489bc8aa448220fc"}
Nov 02 17:51:43.424806 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:43.424734    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6" podStartSLOduration=2.424713719 podStartE2EDuration="2.424713719s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:43.404543335 +0000 UTC m=+1319.895606498" watchObservedRunningTime="2026-11-02 17:51:43.424713719 +0000 UTC m=+1319.915776881"
Nov 02 17:51:44.358807 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:44.358769    4456 generic.go:356] "Generic (PLEG): container finished" podID="05136623-c975-421f-83a8-08c9b8f574e3" containerID="d5a7f9bff5a598255785a777f7ede52ce5db84493e91ffbd489bc8aa448220fc" exitCode=0
Nov 02 17:51:44.359212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:44.358866    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6" event={"ID":"05136623-c975-421f-83a8-08c9b8f574e3","Type":"ContainerDied","Data":"d5a7f9bff5a598255785a777f7ede52ce5db84493e91ffbd489bc8aa448220fc"}
Nov 02 17:51:44.777945 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:44.777916    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"]
Nov 02 17:51:44.805841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:44.805575    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"
Nov 02 17:51:44.820869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:44.818350    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"]
Nov 02 17:51:44.865266 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:44.865228    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c79b51-5464-4227-b6fc-8a16396aa911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c4292c-n8sx5\" (UID: \"a8c79b51-5464-4227-b6fc-8a16396aa911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"
Nov 02 17:51:44.865640 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:44.865618    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hk7s9\" (UniqueName: \"kubernetes.io/projected/a8c79b51-5464-4227-b6fc-8a16396aa911-kube-api-access-hk7s9\") pod \"maas-api-revoke-sub-b0c4292c-n8sx5\" (UID: \"a8c79b51-5464-4227-b6fc-8a16396aa911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"
Nov 02 17:51:44.866049 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:44.866029    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c79b51-5464-4227-b6fc-8a16396aa911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c4292c-n8sx5\" (UID: \"a8c79b51-5464-4227-b6fc-8a16396aa911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"
Nov 02 17:51:44.967521 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:44.967485    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hk7s9\" (UniqueName: \"kubernetes.io/projected/a8c79b51-5464-4227-b6fc-8a16396aa911-kube-api-access-hk7s9\") pod \"maas-api-revoke-sub-b0c4292c-n8sx5\" (UID: \"a8c79b51-5464-4227-b6fc-8a16396aa911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"
Nov 02 17:51:44.967681 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:44.967628    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c79b51-5464-4227-b6fc-8a16396aa911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c4292c-n8sx5\" (UID: \"a8c79b51-5464-4227-b6fc-8a16396aa911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"
Nov 02 17:51:44.967870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:44.967850    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c79b51-5464-4227-b6fc-8a16396aa911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c4292c-n8sx5\" (UID: \"a8c79b51-5464-4227-b6fc-8a16396aa911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"
Nov 02 17:51:44.968339 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:44.968317    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c79b51-5464-4227-b6fc-8a16396aa911-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-b0c4292c-n8sx5\" (UID: \"a8c79b51-5464-4227-b6fc-8a16396aa911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"
Nov 02 17:51:44.971040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:44.971013    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c79b51-5464-4227-b6fc-8a16396aa911-maas-api-xks-ca\") pod \"maas-api-revoke-sub-b0c4292c-n8sx5\" (UID: \"a8c79b51-5464-4227-b6fc-8a16396aa911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"
Nov 02 17:51:45.014595 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.012537    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hk7s9\" (UniqueName: \"kubernetes.io/projected/a8c79b51-5464-4227-b6fc-8a16396aa911-kube-api-access-hk7s9\") pod \"maas-api-revoke-sub-b0c4292c-n8sx5\" (UID: \"a8c79b51-5464-4227-b6fc-8a16396aa911\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"
Nov 02 17:51:45.113459 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.113339    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:45.143782 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.143755    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"
Nov 02 17:51:45.272653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.272594    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:45.555256 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.554735    4456 scope.go:122] "RemoveContainer" containerID="55281d76e5a115ec6a71d8e85c50b4bf336cfd1d51bc2e59b260acf04fb60adb"
Nov 02 17:51:45.647669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.647644    4456 scope.go:122] "RemoveContainer" containerID="e24cc9891fc9fd824c91fbfbff91184fee256eed39286c80691938f48cf2cd25"
Nov 02 17:51:45.658858 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:51:45.655268    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda8c79b51_5464_4227_b6fc_8a16396aa911.slice/crio-f916a1805b4871ae607ba98a7e57c77ebb5971969092d249c972f3c7cbf48104 WatchSource:0}: Error finding container f916a1805b4871ae607ba98a7e57c77ebb5971969092d249c972f3c7cbf48104: Status 404 returned error can't find the container with id f916a1805b4871ae607ba98a7e57c77ebb5971969092d249c972f3c7cbf48104
Nov 02 17:51:45.659794 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.659770    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"]
Nov 02 17:51:45.702111 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.685491    4456 scope.go:122] "RemoveContainer" containerID="97657c79213205657317432093eb88e1146e42258a218a154be8df535ad485e9"
Nov 02 17:51:45.702111 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.701428    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"
Nov 02 17:51:45.734679 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.734656    4456 scope.go:122] "RemoveContainer" containerID="22d60d1711c3fcaa2777f36c5f4f4bc423ed26bbef3e3b1da5fd1d450a05e744"
Nov 02 17:51:45.754525 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.754505    4456 scope.go:122] "RemoveContainer" containerID="6b3136dd6effce815920358b5ef4e25d78f4e6f73f6587a51eadbdc4f33c175d"
Nov 02 17:51:45.840775 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.840740    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/05136623-c975-421f-83a8-08c9b8f574e3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05136623-c975-421f-83a8-08c9b8f574e3-maas-api-xks-ca\") pod \"05136623-c975-421f-83a8-08c9b8f574e3\" (UID: \"05136623-c975-421f-83a8-08c9b8f574e3\") "
Nov 02 17:51:45.840954 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.840825    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/05136623-c975-421f-83a8-08c9b8f574e3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05136623-c975-421f-83a8-08c9b8f574e3-maas-api-openshift-ca\") pod \"05136623-c975-421f-83a8-08c9b8f574e3\" (UID: \"05136623-c975-421f-83a8-08c9b8f574e3\") "
Nov 02 17:51:45.840954 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.840924    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/05136623-c975-421f-83a8-08c9b8f574e3-kube-api-access-2lm8f\" (UniqueName: \"kubernetes.io/projected/05136623-c975-421f-83a8-08c9b8f574e3-kube-api-access-2lm8f\") pod \"05136623-c975-421f-83a8-08c9b8f574e3\" (UID: \"05136623-c975-421f-83a8-08c9b8f574e3\") "
Nov 02 17:51:45.842661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.842619    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/05136623-c975-421f-83a8-08c9b8f574e3-maas-api-openshift-ca" pod "05136623-c975-421f-83a8-08c9b8f574e3" (UID: "05136623-c975-421f-83a8-08c9b8f574e3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:45.844487 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.844462    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/05136623-c975-421f-83a8-08c9b8f574e3-kube-api-access-2lm8f" pod "05136623-c975-421f-83a8-08c9b8f574e3" (UID: "05136623-c975-421f-83a8-08c9b8f574e3"). InnerVolumeSpecName "kube-api-access-2lm8f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:45.845062 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.845035    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/05136623-c975-421f-83a8-08c9b8f574e3-maas-api-xks-ca" pod "05136623-c975-421f-83a8-08c9b8f574e3" (UID: "05136623-c975-421f-83a8-08c9b8f574e3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:51:45.943376 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.943346    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/05136623-c975-421f-83a8-08c9b8f574e3-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:45.943376 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.943373    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/05136623-c975-421f-83a8-08c9b8f574e3-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:45.943376 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:45.943383    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2lm8f\" (UniqueName: \"kubernetes.io/projected/05136623-c975-421f-83a8-08c9b8f574e3-kube-api-access-2lm8f\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:46.278004 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:51:46.277969    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:51:46.456837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:46.456568    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6" event={"ID":"05136623-c975-421f-83a8-08c9b8f574e3","Type":"ContainerDied","Data":"3fb29eed3679aaa7193f5ed33f6fab1fe82d86f6e6dd9566b492c47c26c9ebc0"}
Nov 02 17:51:46.456837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:46.456639    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3fb29eed3679aaa7193f5ed33f6fab1fe82d86f6e6dd9566b492c47c26c9ebc0"
Nov 02 17:51:46.456837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:46.456731    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"
Nov 02 17:51:46.476880 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:46.472474    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5" event={"ID":"a8c79b51-5464-4227-b6fc-8a16396aa911","Type":"ContainerStarted","Data":"9a0a559a6845e4aa2c0fa679f058981c14eda9f74bdce3b25cbc0a31d3b5fa98"}
Nov 02 17:51:46.476880 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:46.472543    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5" event={"ID":"a8c79b51-5464-4227-b6fc-8a16396aa911","Type":"ContainerStarted","Data":"f916a1805b4871ae607ba98a7e57c77ebb5971969092d249c972f3c7cbf48104"}
Nov 02 17:51:46.980009 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:46.979970    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:47.247100 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:47.246742    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"
Nov 02 17:51:47.307630 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:47.307553    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5" podStartSLOduration=3.307531715 podStartE2EDuration="3.307531715s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:44 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:46.532714301 +0000 UTC m=+1323.023777486" watchObservedRunningTime="2026-11-02 17:51:47.307531715 +0000 UTC m=+1323.798594878"
Nov 02 17:51:47.357835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:47.357752    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb" podUID="839ebaa1-25e5-4db7-abdc-7f55f837bff7" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 17:51:47.536909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:47.535757    4456 generic.go:356] "Generic (PLEG): container finished" podID="a8c79b51-5464-4227-b6fc-8a16396aa911" containerID="9a0a559a6845e4aa2c0fa679f058981c14eda9f74bdce3b25cbc0a31d3b5fa98" exitCode=0
Nov 02 17:51:47.536909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:47.536022    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5" event={"ID":"a8c79b51-5464-4227-b6fc-8a16396aa911","Type":"ContainerDied","Data":"9a0a559a6845e4aa2c0fa679f058981c14eda9f74bdce3b25cbc0a31d3b5fa98"}
Nov 02 17:51:48.759188 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:48.759154    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:48.789849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:48.789800    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"
Nov 02 17:51:48.851578 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:48.851501    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a8c79b51-5464-4227-b6fc-8a16396aa911-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c79b51-5464-4227-b6fc-8a16396aa911-maas-api-xks-ca\") pod \"a8c79b51-5464-4227-b6fc-8a16396aa911\" (UID: \"a8c79b51-5464-4227-b6fc-8a16396aa911\") "
Nov 02 17:51:48.851747 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:48.851632    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a8c79b51-5464-4227-b6fc-8a16396aa911-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c79b51-5464-4227-b6fc-8a16396aa911-maas-api-openshift-ca\") pod \"a8c79b51-5464-4227-b6fc-8a16396aa911\" (UID: \"a8c79b51-5464-4227-b6fc-8a16396aa911\") "
Nov 02 17:51:48.851869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:48.851765    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a8c79b51-5464-4227-b6fc-8a16396aa911-kube-api-access-hk7s9\" (UniqueName: \"kubernetes.io/projected/a8c79b51-5464-4227-b6fc-8a16396aa911-kube-api-access-hk7s9\") pod \"a8c79b51-5464-4227-b6fc-8a16396aa911\" (UID: \"a8c79b51-5464-4227-b6fc-8a16396aa911\") "
Nov 02 17:51:48.852259 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:48.852232    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a8c79b51-5464-4227-b6fc-8a16396aa911-maas-api-openshift-ca" pod "a8c79b51-5464-4227-b6fc-8a16396aa911" (UID: "a8c79b51-5464-4227-b6fc-8a16396aa911"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:48.852555 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:48.852533    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a8c79b51-5464-4227-b6fc-8a16396aa911-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:48.860460 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:48.858030    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a8c79b51-5464-4227-b6fc-8a16396aa911-kube-api-access-hk7s9" pod "a8c79b51-5464-4227-b6fc-8a16396aa911" (UID: "a8c79b51-5464-4227-b6fc-8a16396aa911"). InnerVolumeSpecName "kube-api-access-hk7s9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:48.860460 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:48.860027    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a8c79b51-5464-4227-b6fc-8a16396aa911-maas-api-xks-ca" pod "a8c79b51-5464-4227-b6fc-8a16396aa911" (UID: "a8c79b51-5464-4227-b6fc-8a16396aa911"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:51:48.953873 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:48.953831    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hk7s9\" (UniqueName: \"kubernetes.io/projected/a8c79b51-5464-4227-b6fc-8a16396aa911-kube-api-access-hk7s9\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:48.953873 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:48.953863    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a8c79b51-5464-4227-b6fc-8a16396aa911-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:49.558152 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:49.558124    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"
Nov 02 17:51:49.558395 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:49.558123    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5" event={"ID":"a8c79b51-5464-4227-b6fc-8a16396aa911","Type":"ContainerDied","Data":"f916a1805b4871ae607ba98a7e57c77ebb5971969092d249c972f3c7cbf48104"}
Nov 02 17:51:49.558395 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:49.558253    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f916a1805b4871ae607ba98a7e57c77ebb5971969092d249c972f3c7cbf48104"
Nov 02 17:51:50.184054 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.184017    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:50.631432 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.631355    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:50.812842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.812768    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"]
Nov 02 17:51:50.816886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.814213    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a8c79b51-5464-4227-b6fc-8a16396aa911" containerName="revoke-keys"
Nov 02 17:51:50.816886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.814236    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a8c79b51-5464-4227-b6fc-8a16396aa911" containerName="revoke-keys"
Nov 02 17:51:50.816886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.814259    4456 cpu_manager.go:414] "containerMap: removing container" podUID="05136623-c975-421f-83a8-08c9b8f574e3" containerName="revoke-keys"
Nov 02 17:51:50.816886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.814266    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="05136623-c975-421f-83a8-08c9b8f574e3" containerName="revoke-keys"
Nov 02 17:51:50.816886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.814455    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a8c79b51-5464-4227-b6fc-8a16396aa911" containerName="revoke-keys"
Nov 02 17:51:50.816886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.814472    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="05136623-c975-421f-83a8-08c9b8f574e3" containerName="revoke-keys"
Nov 02 17:51:50.839412 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.839381    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:50.846147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.845918    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"]
Nov 02 17:51:50.862500 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.862476    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-a-c3c112-openshift-default-dockercfg-6fgt6\""
Nov 02 17:51:50.888129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.888054    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:50.888300 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.888141    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:50.888300 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.888246    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:50.888446 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.888316    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:50.888446 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.888398    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:50.888567 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.888533    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:50.888630 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.888611    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nqr47\" (UniqueName: \"kubernetes.io/projected/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-kube-api-access-nqr47\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:50.888698 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.888642    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:50.888758 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.888723    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:50.932015 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.931978    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:50.998904 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:50.995896    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.030312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.021882    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.030312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.028766    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.030312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.029195    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.030312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.029314    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.030312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.029415    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.030312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.029578    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.030312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.029666    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nqr47\" (UniqueName: \"kubernetes.io/projected/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-kube-api-access-nqr47\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.030312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.029698    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.030312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.029794    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.030312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.030299    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.034346 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.030537    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.034346 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.030763    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.034346 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.031434    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.104961 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.093474    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:51.133545 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.124958    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.176891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.175284    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.202839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.184194    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.202839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.184847    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nqr47\" (UniqueName: \"kubernetes.io/projected/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-kube-api-access-nqr47\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.326041 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.326002    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:51.472331 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.472299    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:51.475466 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.475423    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:51.583506 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.582011    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"
Nov 02 17:51:51.623339 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.619505    4456 generic.go:356] "Generic (PLEG): container finished" podID="839ebaa1-25e5-4db7-abdc-7f55f837bff7" containerID="648f137883d0ad44adb7ec5c721265dd4b4d1dda41c2a3489d6187de719dab72" exitCode=0
Nov 02 17:51:51.623339 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.619772    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"
Nov 02 17:51:51.643636 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.643596    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb" event={"ID":"839ebaa1-25e5-4db7-abdc-7f55f837bff7","Type":"ContainerDied","Data":"648f137883d0ad44adb7ec5c721265dd4b4d1dda41c2a3489d6187de719dab72"}
Nov 02 17:51:51.643830 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.643666    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb" event={"ID":"839ebaa1-25e5-4db7-abdc-7f55f837bff7","Type":"ContainerDied","Data":"7bc801eec8efb883a7de656646f5f5034a5b335b228f1292da67979c073c5f6d"}
Nov 02 17:51:51.643830 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.643772    4456 scope.go:122] "RemoveContainer" containerID="648f137883d0ad44adb7ec5c721265dd4b4d1dda41c2a3489d6187de719dab72"
Nov 02 17:51:51.652172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.649797    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/839ebaa1-25e5-4db7-abdc-7f55f837bff7-kube-api-access-4nxcw\" (UniqueName: \"kubernetes.io/projected/839ebaa1-25e5-4db7-abdc-7f55f837bff7-kube-api-access-4nxcw\") pod \"839ebaa1-25e5-4db7-abdc-7f55f837bff7\" (UID: \"839ebaa1-25e5-4db7-abdc-7f55f837bff7\") "
Nov 02 17:51:51.652172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.651440    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/839ebaa1-25e5-4db7-abdc-7f55f837bff7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839ebaa1-25e5-4db7-abdc-7f55f837bff7-plugins-config-volume\") pod \"839ebaa1-25e5-4db7-abdc-7f55f837bff7\" (UID: \"839ebaa1-25e5-4db7-abdc-7f55f837bff7\") "
Nov 02 17:51:51.652172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.651926    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/839ebaa1-25e5-4db7-abdc-7f55f837bff7-plugins-config-volume" pod "839ebaa1-25e5-4db7-abdc-7f55f837bff7" (UID: "839ebaa1-25e5-4db7-abdc-7f55f837bff7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:51.652485 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.652438    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/839ebaa1-25e5-4db7-abdc-7f55f837bff7-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:51.653098 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.653053    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/839ebaa1-25e5-4db7-abdc-7f55f837bff7-kube-api-access-4nxcw" pod "839ebaa1-25e5-4db7-abdc-7f55f837bff7" (UID: "839ebaa1-25e5-4db7-abdc-7f55f837bff7"). InnerVolumeSpecName "kube-api-access-4nxcw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:51.663899 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.663789    4456 scope.go:122] "RemoveContainer" containerID="648f137883d0ad44adb7ec5c721265dd4b4d1dda41c2a3489d6187de719dab72"
Nov 02 17:51:51.664313 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:51:51.664258    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"648f137883d0ad44adb7ec5c721265dd4b4d1dda41c2a3489d6187de719dab72\": container with ID starting with 648f137883d0ad44adb7ec5c721265dd4b4d1dda41c2a3489d6187de719dab72 not found: ID does not exist" containerID="648f137883d0ad44adb7ec5c721265dd4b4d1dda41c2a3489d6187de719dab72"
Nov 02 17:51:51.664313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.664283    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"648f137883d0ad44adb7ec5c721265dd4b4d1dda41c2a3489d6187de719dab72"} err="failed to get container status \"648f137883d0ad44adb7ec5c721265dd4b4d1dda41c2a3489d6187de719dab72\": rpc error: code = NotFound desc = could not find container \"648f137883d0ad44adb7ec5c721265dd4b4d1dda41c2a3489d6187de719dab72\": container with ID starting with 648f137883d0ad44adb7ec5c721265dd4b4d1dda41c2a3489d6187de719dab72 not found: ID does not exist"
Nov 02 17:51:51.754374 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.754335    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4nxcw\" (UniqueName: \"kubernetes.io/projected/839ebaa1-25e5-4db7-abdc-7f55f837bff7-kube-api-access-4nxcw\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:51.839213 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.839184    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"]
Nov 02 17:51:51.840255 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:51:51.840233    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3f8f4a9_e906_4bb8_a6e6_f985fa71c42e.slice/crio-8830dff6831e37d283ecd97e0dc6c3ed4b41844695e566a3d55e22aacc3a9f49 WatchSource:0}: Error finding container 8830dff6831e37d283ecd97e0dc6c3ed4b41844695e566a3d55e22aacc3a9f49: Status 404 returned error can't find the container with id 8830dff6831e37d283ecd97e0dc6c3ed4b41844695e566a3d55e22aacc3a9f49
Nov 02 17:51:51.843177 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.843145    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:51.843263 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.843208    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:51.843263 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.843235    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:51.946210 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.946181    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"]
Nov 02 17:51:51.978012 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.977980    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"]
Nov 02 17:51:51.984892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.984448    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-t7snb"]
Nov 02 17:51:51.994102 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:51.994071    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"]
Nov 02 17:51:52.153244 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:52.152662    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:52.326154 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:52.326018    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:52.630885 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:52.630780    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp" event={"ID":"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e","Type":"ContainerStarted","Data":"b53a79d0493a360d98414bc8e408bb9b56548a39e73e6df6cfc4f94d5d63f132"}
Nov 02 17:51:52.630885 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:52.630853    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp" event={"ID":"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e","Type":"ContainerStarted","Data":"8830dff6831e37d283ecd97e0dc6c3ed4b41844695e566a3d55e22aacc3a9f49"}
Nov 02 17:51:52.645331 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:52.636251    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:52.680473 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:52.679189    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp" podStartSLOduration=2.679173148 podStartE2EDuration="2.679173148s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:51:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:51:52.677197879 +0000 UTC m=+1329.168261051" watchObservedRunningTime="2026-11-02 17:51:52.679173148 +0000 UTC m=+1329.170236384"
Nov 02 17:51:53.476135 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:53.476105    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:53.482879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:53.482798    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:53.619266 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:53.619226    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="839ebaa1-25e5-4db7-abdc-7f55f837bff7" path="/var/lib/kubelet/pods/839ebaa1-25e5-4db7-abdc-7f55f837bff7/volumes"
Nov 02 17:51:53.641415 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:53.641385    4456 generic.go:356] "Generic (PLEG): container finished" podID="92a4895f-f62c-4012-8114-49a4e97a8644" containerID="573cf1c3514bb6d994f4e99ee4466848b20f08591ecb7d4646a422ba5b8684aa" exitCode=0
Nov 02 17:51:53.641575 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:53.641455    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr" event={"ID":"92a4895f-f62c-4012-8114-49a4e97a8644","Type":"ContainerDied","Data":"573cf1c3514bb6d994f4e99ee4466848b20f08591ecb7d4646a422ba5b8684aa"}
Nov 02 17:51:53.641874 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:53.641845    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:53.642048 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:53.642028    4456 scope.go:122] "RemoveContainer" containerID="573cf1c3514bb6d994f4e99ee4466848b20f08591ecb7d4646a422ba5b8684aa"
Nov 02 17:51:53.643070 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:53.642755    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:51:53.643881 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:53.643856    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:53.643934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:53.643923    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:53.643973 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:53.643954    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:51:54.555840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:54.553293    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:54.666835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:54.666778    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr" event={"ID":"92a4895f-f62c-4012-8114-49a4e97a8644","Type":"ContainerStarted","Data":"28b2767f02ab7967606653f3fb185bb68f6a2fc0cd525c98bffcc4b47c14c6c0"}
Nov 02 17:51:55.230672 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:55.230636    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:55.357980 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:55.357950    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:51:55.359190 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:55.359161    4456 patch_prober.go:29] interesting pod/e2e-api-b-769b70-openshift-default-897676bcc-gfczr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 17:51:55.359312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:55.359212    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr" podUID="92a4895f-f62c-4012-8114-49a4e97a8644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 17:51:55.414644 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:55.414609    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:56.168717 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:56.168681    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"]
Nov 02 17:51:56.169232 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:56.168934    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p" podUID="edc728ac-8cd4-44e7-b831-8bbc21a40db0" containerName="maas-api" containerID="cri-o://4bcac2cac206fef1426452fcac125a7e35c994f98f1470b119a94ed2de318f0c" gracePeriod=30
Nov 02 17:51:56.254244 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:56.254204    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:56.358682 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:56.358143    4456 patch_prober.go:29] interesting pod/e2e-api-b-769b70-openshift-default-897676bcc-gfczr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 17:51:56.358682 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:56.358217    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr" podUID="92a4895f-f62c-4012-8114-49a4e97a8644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 17:51:56.431890 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:56.414676    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:56.457670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:56.452244    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:51:56.912659 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:56.912528    4456 generic.go:356] "Generic (PLEG): container finished" podID="edc728ac-8cd4-44e7-b831-8bbc21a40db0" containerID="4bcac2cac206fef1426452fcac125a7e35c994f98f1470b119a94ed2de318f0c" exitCode=0
Nov 02 17:51:56.914246 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:56.914123    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p" event={"ID":"edc728ac-8cd4-44e7-b831-8bbc21a40db0","Type":"ContainerDied","Data":"4bcac2cac206fef1426452fcac125a7e35c994f98f1470b119a94ed2de318f0c"}
Nov 02 17:51:56.974848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:56.963595    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr" podUID="b3bc4897-c112-464b-8072-62554dcc476b" containerName="payload-processing" containerID="cri-o://2cdf2586ed8874a81d7034e27cd512ad9c75703935d263c4a2cf909c69ea362b" gracePeriod=25
Nov 02 17:51:57.042838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.042689    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr" podUID="2612d124-cc29-46af-bad1-f8af29bee00f" containerName="payload-pre-processing" containerID="cri-o://80c4276055304e92ba37840b9e20ba31285ac65a521903b6cb1c0f3c44a44077" gracePeriod=25
Nov 02 17:51:57.161879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.160782    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:57.179621 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.179571    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"]
Nov 02 17:51:57.261872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.259245    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-0d1f5d84-6tbx6"]
Nov 02 17:51:57.336705 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.333294    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:57.344443 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.344416    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:51:57.358158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.358126    4456 patch_prober.go:29] interesting pod/e2e-api-b-769b70-openshift-default-897676bcc-gfczr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 17:51:57.358346 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.358187    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr" podUID="92a4895f-f62c-4012-8114-49a4e97a8644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 17:51:57.429274 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.428439    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/edc728ac-8cd4-44e7-b831-8bbc21a40db0-kube-api-access-g8mkb\" (UniqueName: \"kubernetes.io/projected/edc728ac-8cd4-44e7-b831-8bbc21a40db0-kube-api-access-g8mkb\") pod \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") "
Nov 02 17:51:57.429274 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.428556    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/edc728ac-8cd4-44e7-b831-8bbc21a40db0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc728ac-8cd4-44e7-b831-8bbc21a40db0-openshift-service-ca\") pod \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") "
Nov 02 17:51:57.429274 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.428636    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edc728ac-8cd4-44e7-b831-8bbc21a40db0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc728ac-8cd4-44e7-b831-8bbc21a40db0-maas-api-tls\") pod \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") "
Nov 02 17:51:57.429274 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.428754    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/edc728ac-8cd4-44e7-b831-8bbc21a40db0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc728ac-8cd4-44e7-b831-8bbc21a40db0-metrics-certs\") pod \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\" (UID: \"edc728ac-8cd4-44e7-b831-8bbc21a40db0\") "
Nov 02 17:51:57.433996 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.430450    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/edc728ac-8cd4-44e7-b831-8bbc21a40db0-openshift-service-ca" pod "edc728ac-8cd4-44e7-b831-8bbc21a40db0" (UID: "edc728ac-8cd4-44e7-b831-8bbc21a40db0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:57.433996 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.433955    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/edc728ac-8cd4-44e7-b831-8bbc21a40db0-kube-api-access-g8mkb" pod "edc728ac-8cd4-44e7-b831-8bbc21a40db0" (UID: "edc728ac-8cd4-44e7-b831-8bbc21a40db0"). InnerVolumeSpecName "kube-api-access-g8mkb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:57.438023 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.436248    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc728ac-8cd4-44e7-b831-8bbc21a40db0-maas-api-tls" pod "edc728ac-8cd4-44e7-b831-8bbc21a40db0" (UID: "edc728ac-8cd4-44e7-b831-8bbc21a40db0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:51:57.438023 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.436301    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/edc728ac-8cd4-44e7-b831-8bbc21a40db0-metrics-certs" pod "edc728ac-8cd4-44e7-b831-8bbc21a40db0" (UID: "edc728ac-8cd4-44e7-b831-8bbc21a40db0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:51:57.535063 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.535007    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g8mkb\" (UniqueName: \"kubernetes.io/projected/edc728ac-8cd4-44e7-b831-8bbc21a40db0-kube-api-access-g8mkb\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:57.535063 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.535055    4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/edc728ac-8cd4-44e7-b831-8bbc21a40db0-openshift-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:57.535313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.535072    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/edc728ac-8cd4-44e7-b831-8bbc21a40db0-maas-api-tls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:57.535313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.535094    4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/edc728ac-8cd4-44e7-b831-8bbc21a40db0-metrics-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:57.625431 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.625355    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:57.643963 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.643925    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="05136623-c975-421f-83a8-08c9b8f574e3" path="/var/lib/kubelet/pods/05136623-c975-421f-83a8-08c9b8f574e3/volumes"
Nov 02 17:51:57.650259 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.650236    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"
Nov 02 17:51:57.678968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.678942    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"
Nov 02 17:51:57.842839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.839788    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2612d124-cc29-46af-bad1-f8af29bee00f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2612d124-cc29-46af-bad1-f8af29bee00f-plugins-config-volume\") pod \"2612d124-cc29-46af-bad1-f8af29bee00f\" (UID: \"2612d124-cc29-46af-bad1-f8af29bee00f\") "
Nov 02 17:51:57.842839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.839856    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b3bc4897-c112-464b-8072-62554dcc476b-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3bc4897-c112-464b-8072-62554dcc476b-plugins-config-volume\") pod \"b3bc4897-c112-464b-8072-62554dcc476b\" (UID: \"b3bc4897-c112-464b-8072-62554dcc476b\") "
Nov 02 17:51:57.842839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.840313    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b3bc4897-c112-464b-8072-62554dcc476b-kube-api-access-6c2ls\" (UniqueName: \"kubernetes.io/projected/b3bc4897-c112-464b-8072-62554dcc476b-kube-api-access-6c2ls\") pod \"b3bc4897-c112-464b-8072-62554dcc476b\" (UID: \"b3bc4897-c112-464b-8072-62554dcc476b\") "
Nov 02 17:51:57.842839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.840476    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2612d124-cc29-46af-bad1-f8af29bee00f-kube-api-access-hck44\" (UniqueName: \"kubernetes.io/projected/2612d124-cc29-46af-bad1-f8af29bee00f-kube-api-access-hck44\") pod \"2612d124-cc29-46af-bad1-f8af29bee00f\" (UID: \"2612d124-cc29-46af-bad1-f8af29bee00f\") "
Nov 02 17:51:57.870123 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.870059    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b3bc4897-c112-464b-8072-62554dcc476b-plugins-config-volume" pod "b3bc4897-c112-464b-8072-62554dcc476b" (UID: "b3bc4897-c112-464b-8072-62554dcc476b"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:57.870838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.870792    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2612d124-cc29-46af-bad1-f8af29bee00f-plugins-config-volume" pod "2612d124-cc29-46af-bad1-f8af29bee00f" (UID: "2612d124-cc29-46af-bad1-f8af29bee00f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:51:57.907930 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.907798    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b3bc4897-c112-464b-8072-62554dcc476b-kube-api-access-6c2ls" pod "b3bc4897-c112-464b-8072-62554dcc476b" (UID: "b3bc4897-c112-464b-8072-62554dcc476b"). InnerVolumeSpecName "kube-api-access-6c2ls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:57.924904 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.912233    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:57.955848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.950233    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/2612d124-cc29-46af-bad1-f8af29bee00f-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:57.962846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.956485    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b3bc4897-c112-464b-8072-62554dcc476b-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:57.962846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:57.956515    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6c2ls\" (UniqueName: \"kubernetes.io/projected/b3bc4897-c112-464b-8072-62554dcc476b-kube-api-access-6c2ls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:58.012422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.012385    4456 generic.go:356] "Generic (PLEG): container finished" podID="2612d124-cc29-46af-bad1-f8af29bee00f" containerID="80c4276055304e92ba37840b9e20ba31285ac65a521903b6cb1c0f3c44a44077" exitCode=0
Nov 02 17:51:58.012750 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.012719    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr" event={"ID":"2612d124-cc29-46af-bad1-f8af29bee00f","Type":"ContainerDied","Data":"80c4276055304e92ba37840b9e20ba31285ac65a521903b6cb1c0f3c44a44077"}
Nov 02 17:51:58.013876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.012955    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr" event={"ID":"2612d124-cc29-46af-bad1-f8af29bee00f","Type":"ContainerDied","Data":"7fb75d74f8fa337c083e869c783a7ae20c15153f051ee26b17392edde937c6ce"}
Nov 02 17:51:58.014059 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.014041    4456 scope.go:122] "RemoveContainer" containerID="80c4276055304e92ba37840b9e20ba31285ac65a521903b6cb1c0f3c44a44077"
Nov 02 17:51:58.014356 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.014338    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"
Nov 02 17:51:58.030905 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.027990    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2612d124-cc29-46af-bad1-f8af29bee00f-kube-api-access-hck44" pod "2612d124-cc29-46af-bad1-f8af29bee00f" (UID: "2612d124-cc29-46af-bad1-f8af29bee00f"). InnerVolumeSpecName "kube-api-access-hck44". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:51:58.056792 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.056745    4456 generic.go:356] "Generic (PLEG): container finished" podID="b3bc4897-c112-464b-8072-62554dcc476b" containerID="2cdf2586ed8874a81d7034e27cd512ad9c75703935d263c4a2cf909c69ea362b" exitCode=0
Nov 02 17:51:58.057246 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.057204    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr" event={"ID":"b3bc4897-c112-464b-8072-62554dcc476b","Type":"ContainerDied","Data":"2cdf2586ed8874a81d7034e27cd512ad9c75703935d263c4a2cf909c69ea362b"}
Nov 02 17:51:58.057483 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.057449    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr" event={"ID":"b3bc4897-c112-464b-8072-62554dcc476b","Type":"ContainerDied","Data":"0da986c6d24244bf8e79849f034ba5bfb7c2a0f705aca10bcd16b8689c2fdd94"}
Nov 02 17:51:58.057696 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.057671    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"
Nov 02 17:51:58.080852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.078308    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hck44\" (UniqueName: \"kubernetes.io/projected/2612d124-cc29-46af-bad1-f8af29bee00f-kube-api-access-hck44\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:51:58.158320 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.149472    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p" event={"ID":"edc728ac-8cd4-44e7-b831-8bbc21a40db0","Type":"ContainerDied","Data":"9ffe0e4ce8b8f0f099a5f2067a0d8208b4efbc1758e849d41e437213a80c80de"}
Nov 02 17:51:58.158320 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.149615    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"
Nov 02 17:51:58.222642 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.218042    4456 scope.go:122] "RemoveContainer" containerID="80c4276055304e92ba37840b9e20ba31285ac65a521903b6cb1c0f3c44a44077"
Nov 02 17:51:58.222642 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:51:58.218550    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80c4276055304e92ba37840b9e20ba31285ac65a521903b6cb1c0f3c44a44077\": container with ID starting with 80c4276055304e92ba37840b9e20ba31285ac65a521903b6cb1c0f3c44a44077 not found: ID does not exist" containerID="80c4276055304e92ba37840b9e20ba31285ac65a521903b6cb1c0f3c44a44077"
Nov 02 17:51:58.222642 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.218595    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80c4276055304e92ba37840b9e20ba31285ac65a521903b6cb1c0f3c44a44077"} err="failed to get container status \"80c4276055304e92ba37840b9e20ba31285ac65a521903b6cb1c0f3c44a44077\": rpc error: code = NotFound desc = could not find container \"80c4276055304e92ba37840b9e20ba31285ac65a521903b6cb1c0f3c44a44077\": container with ID starting with 80c4276055304e92ba37840b9e20ba31285ac65a521903b6cb1c0f3c44a44077 not found: ID does not exist"
Nov 02 17:51:58.222642 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.218620    4456 scope.go:122] "RemoveContainer" containerID="2cdf2586ed8874a81d7034e27cd512ad9c75703935d263c4a2cf909c69ea362b"
Nov 02 17:51:58.269378 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.268636    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"]
Nov 02 17:51:58.272315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.272289    4456 scope.go:122] "RemoveContainer" containerID="2cdf2586ed8874a81d7034e27cd512ad9c75703935d263c4a2cf909c69ea362b"
Nov 02 17:51:58.276058 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.276025    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-mt-0d1f5d84-569d8694f6-zdq7p"]
Nov 02 17:51:58.277355 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.277333    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"]
Nov 02 17:51:58.278275 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:51:58.278254    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2cdf2586ed8874a81d7034e27cd512ad9c75703935d263c4a2cf909c69ea362b\": container with ID starting with 2cdf2586ed8874a81d7034e27cd512ad9c75703935d263c4a2cf909c69ea362b not found: ID does not exist" containerID="2cdf2586ed8874a81d7034e27cd512ad9c75703935d263c4a2cf909c69ea362b"
Nov 02 17:51:58.278375 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.278294    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2cdf2586ed8874a81d7034e27cd512ad9c75703935d263c4a2cf909c69ea362b"} err="failed to get container status \"2cdf2586ed8874a81d7034e27cd512ad9c75703935d263c4a2cf909c69ea362b\": rpc error: code = NotFound desc = could not find container \"2cdf2586ed8874a81d7034e27cd512ad9c75703935d263c4a2cf909c69ea362b\": container with ID starting with 2cdf2586ed8874a81d7034e27cd512ad9c75703935d263c4a2cf909c69ea362b not found: ID does not exist"
Nov 02 17:51:58.278375 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.278321    4456 scope.go:122] "RemoveContainer" containerID="4bcac2cac206fef1426452fcac125a7e35c994f98f1470b119a94ed2de318f0c"
Nov 02 17:51:58.285153 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.285131    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-mt-0d1f5d84-f5d99b75-pfqzr"]
Nov 02 17:51:58.358061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.358026    4456 patch_prober.go:29] interesting pod/e2e-api-b-769b70-openshift-default-897676bcc-gfczr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 17:51:58.358215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.358081    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr" podUID="92a4895f-f62c-4012-8114-49a4e97a8644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 17:51:58.372294 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.372265    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"]
Nov 02 17:51:58.376911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:58.376884    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-0d1f5d84-568d6fc7f9-fgvzr"]
Nov 02 17:51:59.276537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:59.276495    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:51:59.368875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:59.367216    4456 patch_prober.go:29] interesting pod/e2e-api-b-769b70-openshift-default-897676bcc-gfczr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 17:51:59.368875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:59.367279    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr" podUID="92a4895f-f62c-4012-8114-49a4e97a8644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 17:51:59.623793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:59.623722    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2612d124-cc29-46af-bad1-f8af29bee00f" path="/var/lib/kubelet/pods/2612d124-cc29-46af-bad1-f8af29bee00f/volumes"
Nov 02 17:51:59.624205 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:59.624190    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b3bc4897-c112-464b-8072-62554dcc476b" path="/var/lib/kubelet/pods/b3bc4897-c112-464b-8072-62554dcc476b/volumes"
Nov 02 17:51:59.624616 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:51:59.624604    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="edc728ac-8cd4-44e7-b831-8bbc21a40db0" path="/var/lib/kubelet/pods/edc728ac-8cd4-44e7-b831-8bbc21a40db0/volumes"
Nov 02 17:52:00.357402 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:00.357367    4456 patch_prober.go:29] interesting pod/e2e-api-b-769b70-openshift-default-897676bcc-gfczr container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused" start-of-body=
Nov 02 17:52:00.357800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:00.357421    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr" podUID="92a4895f-f62c-4012-8114-49a4e97a8644" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.143:15021/healthz/ready\": dial tcp 10.134.0.143:15021: connect: connection refused"
Nov 02 17:52:01.141510 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:01.141476    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:01.349582 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:01.349535    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:01.362898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:01.362870    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:52:01.364041 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:01.364024    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:52:03.068252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:03.068221    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:04.697606 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:04.697565    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:04.849532 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:04.849497    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:05.027137 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:05.027054    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:05.034041 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:05.034008    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"]
Nov 02 17:52:05.193861 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:05.189426    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:05.251192 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:05.251150    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4" podUID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerName="istio-proxy" containerID="cri-o://24ae40e543016679f95f4771c3038102f64f7638708d8e314fe7c78ac01479fc" gracePeriod=30
Nov 02 17:52:06.069399 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:06.069364    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:06.240566 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:06.240531    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:06.927017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:06.926973    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:07.360859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.360806    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:07.430616 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.430576    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"]
Nov 02 17:52:07.431433 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.431411    4456 cpu_manager.go:414] "containerMap: removing container" podUID="839ebaa1-25e5-4db7-abdc-7f55f837bff7" containerName="payload-pre-processing"
Nov 02 17:52:07.431433 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.431433    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="839ebaa1-25e5-4db7-abdc-7f55f837bff7" containerName="payload-pre-processing"
Nov 02 17:52:07.431614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.431453    4456 cpu_manager.go:414] "containerMap: removing container" podUID="edc728ac-8cd4-44e7-b831-8bbc21a40db0" containerName="maas-api"
Nov 02 17:52:07.431614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.431461    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="edc728ac-8cd4-44e7-b831-8bbc21a40db0" containerName="maas-api"
Nov 02 17:52:07.431614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.431497    4456 cpu_manager.go:414] "containerMap: removing container" podUID="b3bc4897-c112-464b-8072-62554dcc476b" containerName="payload-processing"
Nov 02 17:52:07.431614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.431504    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b3bc4897-c112-464b-8072-62554dcc476b" containerName="payload-processing"
Nov 02 17:52:07.431614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.431559    4456 cpu_manager.go:414] "containerMap: removing container" podUID="2612d124-cc29-46af-bad1-f8af29bee00f" containerName="payload-pre-processing"
Nov 02 17:52:07.431614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.431566    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2612d124-cc29-46af-bad1-f8af29bee00f" containerName="payload-pre-processing"
Nov 02 17:52:07.431975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.431759    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="839ebaa1-25e5-4db7-abdc-7f55f837bff7" containerName="payload-pre-processing"
Nov 02 17:52:07.431975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.431774    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="edc728ac-8cd4-44e7-b831-8bbc21a40db0" containerName="maas-api"
Nov 02 17:52:07.431975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.431789    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b3bc4897-c112-464b-8072-62554dcc476b" containerName="payload-processing"
Nov 02 17:52:07.431975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.431810    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="2612d124-cc29-46af-bad1-f8af29bee00f" containerName="payload-pre-processing"
Nov 02 17:52:07.435756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.435735    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"
Nov 02 17:52:07.455779 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.455752    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"]
Nov 02 17:52:07.462866 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.462841    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:52:07.463002 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.462911    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:52:07.531415 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.531381    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-czdlr\" (UniqueName: \"kubernetes.io/projected/821784b6-5f58-4ce6-845e-50077f332e29-kube-api-access-czdlr\") pod \"maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p\" (UID: \"821784b6-5f58-4ce6-845e-50077f332e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"
Nov 02 17:52:07.531603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.531513    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/821784b6-5f58-4ce6-845e-50077f332e29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p\" (UID: \"821784b6-5f58-4ce6-845e-50077f332e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"
Nov 02 17:52:07.531780 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.531763    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/821784b6-5f58-4ce6-845e-50077f332e29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p\" (UID: \"821784b6-5f58-4ce6-845e-50077f332e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"
Nov 02 17:52:07.566706 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.566671    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:07.633250 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.633180    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/821784b6-5f58-4ce6-845e-50077f332e29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p\" (UID: \"821784b6-5f58-4ce6-845e-50077f332e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"
Nov 02 17:52:07.633433 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.633374    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/821784b6-5f58-4ce6-845e-50077f332e29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p\" (UID: \"821784b6-5f58-4ce6-845e-50077f332e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"
Nov 02 17:52:07.633488 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.633471    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-czdlr\" (UniqueName: \"kubernetes.io/projected/821784b6-5f58-4ce6-845e-50077f332e29-kube-api-access-czdlr\") pod \"maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p\" (UID: \"821784b6-5f58-4ce6-845e-50077f332e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"
Nov 02 17:52:07.634033 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.633994    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/821784b6-5f58-4ce6-845e-50077f332e29-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p\" (UID: \"821784b6-5f58-4ce6-845e-50077f332e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"
Nov 02 17:52:07.635874 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.635852    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/821784b6-5f58-4ce6-845e-50077f332e29-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p\" (UID: \"821784b6-5f58-4ce6-845e-50077f332e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"
Nov 02 17:52:07.657776 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.657747    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-czdlr\" (UniqueName: \"kubernetes.io/projected/821784b6-5f58-4ce6-845e-50077f332e29-kube-api-access-czdlr\") pod \"maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p\" (UID: \"821784b6-5f58-4ce6-845e-50077f332e29\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"
Nov 02 17:52:07.732738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.732699    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:07.752929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:07.752901    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"
Nov 02 17:52:08.072679 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:08.072652    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"]
Nov 02 17:52:08.073891 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:52:08.073858    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod821784b6_5f58_4ce6_845e_50077f332e29.slice/crio-a053b7646e243cfaf22e33dab2e8c674f88d56a9a2f2c725545b9bea9e0f4d9b WatchSource:0}: Error finding container a053b7646e243cfaf22e33dab2e8c674f88d56a9a2f2c725545b9bea9e0f4d9b: Status 404 returned error can't find the container with id a053b7646e243cfaf22e33dab2e8c674f88d56a9a2f2c725545b9bea9e0f4d9b
Nov 02 17:52:08.279487 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:08.279445    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p" event={"ID":"821784b6-5f58-4ce6-845e-50077f332e29","Type":"ContainerStarted","Data":"a053b7646e243cfaf22e33dab2e8c674f88d56a9a2f2c725545b9bea9e0f4d9b"}
Nov 02 17:52:08.973312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:08.973249    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:09.135914 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.135886    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"]
Nov 02 17:52:09.140688 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.140671    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"
Nov 02 17:52:09.158681 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.158649    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"]
Nov 02 17:52:09.176145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.176116    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gg767\" (UniqueName: \"kubernetes.io/projected/6016b480-9073-45a6-b00b-84d4ab57d67b-kube-api-access-gg767\") pod \"maas-api-revoke-sub-c4d48df1-xs6xh\" (UID: \"6016b480-9073-45a6-b00b-84d4ab57d67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"
Nov 02 17:52:09.176431 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.176409    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6016b480-9073-45a6-b00b-84d4ab57d67b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4d48df1-xs6xh\" (UID: \"6016b480-9073-45a6-b00b-84d4ab57d67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"
Nov 02 17:52:09.176529 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.176481    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6016b480-9073-45a6-b00b-84d4ab57d67b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4d48df1-xs6xh\" (UID: \"6016b480-9073-45a6-b00b-84d4ab57d67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"
Nov 02 17:52:09.278019 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.277940    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6016b480-9073-45a6-b00b-84d4ab57d67b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4d48df1-xs6xh\" (UID: \"6016b480-9073-45a6-b00b-84d4ab57d67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"
Nov 02 17:52:09.278176 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.278121    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gg767\" (UniqueName: \"kubernetes.io/projected/6016b480-9073-45a6-b00b-84d4ab57d67b-kube-api-access-gg767\") pod \"maas-api-revoke-sub-c4d48df1-xs6xh\" (UID: \"6016b480-9073-45a6-b00b-84d4ab57d67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"
Nov 02 17:52:09.278287 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.278262    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6016b480-9073-45a6-b00b-84d4ab57d67b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4d48df1-xs6xh\" (UID: \"6016b480-9073-45a6-b00b-84d4ab57d67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"
Nov 02 17:52:09.278705 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.278677    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6016b480-9073-45a6-b00b-84d4ab57d67b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c4d48df1-xs6xh\" (UID: \"6016b480-9073-45a6-b00b-84d4ab57d67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"
Nov 02 17:52:09.281181 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.281156    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6016b480-9073-45a6-b00b-84d4ab57d67b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c4d48df1-xs6xh\" (UID: \"6016b480-9073-45a6-b00b-84d4ab57d67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"
Nov 02 17:52:09.290364 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.290336    4456 generic.go:356] "Generic (PLEG): container finished" podID="821784b6-5f58-4ce6-845e-50077f332e29" containerID="49a65df5ba36041e13d2bbcbed7821d59f5a280bcda603271fc0845d2985dd6a" exitCode=0
Nov 02 17:52:09.290619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.290370    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p" event={"ID":"821784b6-5f58-4ce6-845e-50077f332e29","Type":"ContainerDied","Data":"49a65df5ba36041e13d2bbcbed7821d59f5a280bcda603271fc0845d2985dd6a"}
Nov 02 17:52:09.301859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.301838    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gg767\" (UniqueName: \"kubernetes.io/projected/6016b480-9073-45a6-b00b-84d4ab57d67b-kube-api-access-gg767\") pod \"maas-api-revoke-sub-c4d48df1-xs6xh\" (UID: \"6016b480-9073-45a6-b00b-84d4ab57d67b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"
Nov 02 17:52:09.490411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.490370    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"
Nov 02 17:52:09.810287 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:09.810247    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"]
Nov 02 17:52:09.810939 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:52:09.810908    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6016b480_9073_45a6_b00b_84d4ab57d67b.slice/crio-d28d7ff8a800f88bdc7201130b065251cce8c97f4c4e61047376a1de2c9e295b WatchSource:0}: Error finding container d28d7ff8a800f88bdc7201130b065251cce8c97f4c4e61047376a1de2c9e295b: Status 404 returned error can't find the container with id d28d7ff8a800f88bdc7201130b065251cce8c97f4c4e61047376a1de2c9e295b
Nov 02 17:52:10.306196 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.306117    4456 generic.go:356] "Generic (PLEG): container finished" podID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerID="24ae40e543016679f95f4771c3038102f64f7638708d8e314fe7c78ac01479fc" exitCode=0
Nov 02 17:52:10.307490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.306261    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4" event={"ID":"2268303a-b8a6-4783-b4fc-3467cd0c2bb3","Type":"ContainerDied","Data":"24ae40e543016679f95f4771c3038102f64f7638708d8e314fe7c78ac01479fc"}
Nov 02 17:52:10.307490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.306334    4456 scope.go:122] "RemoveContainer" containerID="32c5e815dd3a88821539f897b9d64cab531b39e5121bc42aa66691c79ec0ae1b"
Nov 02 17:52:10.313646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.312436    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh" event={"ID":"6016b480-9073-45a6-b00b-84d4ab57d67b","Type":"ContainerStarted","Data":"d28d7ff8a800f88bdc7201130b065251cce8c97f4c4e61047376a1de2c9e295b"}
Nov 02 17:52:10.358810 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.358745    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh" podStartSLOduration=1.358724614 podStartE2EDuration="1.358724614s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:10.355994601 +0000 UTC m=+1346.847057788" watchObservedRunningTime="2026-11-02 17:52:10.358724614 +0000 UTC m=+1346.849787775"
Nov 02 17:52:10.505951 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.505918    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:52:10.565231 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.564918    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"
Nov 02 17:52:10.619885 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.619854    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-envoy\") pod \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") "
Nov 02 17:52:10.620078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.619921    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/821784b6-5f58-4ce6-845e-50077f332e29-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/821784b6-5f58-4ce6-845e-50077f332e29-maas-api-openshift-ca\") pod \"821784b6-5f58-4ce6-845e-50077f332e29\" (UID: \"821784b6-5f58-4ce6-845e-50077f332e29\") "
Nov 02 17:52:10.620169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.620150    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-workload-certs\") pod \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") "
Nov 02 17:52:10.620236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.620219    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/821784b6-5f58-4ce6-845e-50077f332e29-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/821784b6-5f58-4ce6-845e-50077f332e29-maas-api-xks-ca\") pod \"821784b6-5f58-4ce6-845e-50077f332e29\" (UID: \"821784b6-5f58-4ce6-845e-50077f332e29\") "
Nov 02 17:52:10.620305 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.620250    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-credential-socket\") pod \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") "
Nov 02 17:52:10.620369 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.620352    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/821784b6-5f58-4ce6-845e-50077f332e29-kube-api-access-czdlr\" (UniqueName: \"kubernetes.io/projected/821784b6-5f58-4ce6-845e-50077f332e29-kube-api-access-czdlr\") pod \"821784b6-5f58-4ce6-845e-50077f332e29\" (UID: \"821784b6-5f58-4ce6-845e-50077f332e29\") "
Nov 02 17:52:10.620426 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.620405    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-data\") pod \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") "
Nov 02 17:52:10.620530 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.620521    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istiod-ca-cert\") pod \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") "
Nov 02 17:52:10.620593 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.620552    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-workload-socket\") pod \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") "
Nov 02 17:52:10.620655 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.620598    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-kube-api-access-w6dvp\" (UniqueName: \"kubernetes.io/projected/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-kube-api-access-w6dvp\") pod \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") "
Nov 02 17:52:10.620711 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.620656    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-token\" (UniqueName: \"kubernetes.io/projected/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-token\") pod \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") "
Nov 02 17:52:10.620711 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.620660    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-workload-certs" pod "2268303a-b8a6-4783-b4fc-3467cd0c2bb3" (UID: "2268303a-b8a6-4783-b4fc-3467cd0c2bb3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:10.620711 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.620696    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-podinfo\") pod \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\" (UID: \"2268303a-b8a6-4783-b4fc-3467cd0c2bb3\") "
Nov 02 17:52:10.621775 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.621445    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-credential-socket" pod "2268303a-b8a6-4783-b4fc-3467cd0c2bb3" (UID: "2268303a-b8a6-4783-b4fc-3467cd0c2bb3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:10.621775 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.621571    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-workload-socket" pod "2268303a-b8a6-4783-b4fc-3467cd0c2bb3" (UID: "2268303a-b8a6-4783-b4fc-3467cd0c2bb3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:10.621775 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.621638    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:10.621775 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.621659    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:10.622166 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.621908    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istiod-ca-cert" pod "2268303a-b8a6-4783-b4fc-3467cd0c2bb3" (UID: "2268303a-b8a6-4783-b4fc-3467cd0c2bb3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:10.622411 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.622242    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-data" pod "2268303a-b8a6-4783-b4fc-3467cd0c2bb3" (UID: "2268303a-b8a6-4783-b4fc-3467cd0c2bb3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:10.623041 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.623018    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/821784b6-5f58-4ce6-845e-50077f332e29-maas-api-openshift-ca" pod "821784b6-5f58-4ce6-845e-50077f332e29" (UID: "821784b6-5f58-4ce6-845e-50077f332e29"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:10.627235 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.627202    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/821784b6-5f58-4ce6-845e-50077f332e29-maas-api-xks-ca" pod "821784b6-5f58-4ce6-845e-50077f332e29" (UID: "821784b6-5f58-4ce6-845e-50077f332e29"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:52:10.627358 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.627302    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-token" pod "2268303a-b8a6-4783-b4fc-3467cd0c2bb3" (UID: "2268303a-b8a6-4783-b4fc-3467cd0c2bb3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:10.627448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.627428    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-kube-api-access-w6dvp" pod "2268303a-b8a6-4783-b4fc-3467cd0c2bb3" (UID: "2268303a-b8a6-4783-b4fc-3467cd0c2bb3"). InnerVolumeSpecName "kube-api-access-w6dvp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:10.627555 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.627537    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-envoy" pod "2268303a-b8a6-4783-b4fc-3467cd0c2bb3" (UID: "2268303a-b8a6-4783-b4fc-3467cd0c2bb3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:52:10.627795 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.627780    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-podinfo" pod "2268303a-b8a6-4783-b4fc-3467cd0c2bb3" (UID: "2268303a-b8a6-4783-b4fc-3467cd0c2bb3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:52:10.629150 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.629121    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/821784b6-5f58-4ce6-845e-50077f332e29-kube-api-access-czdlr" pod "821784b6-5f58-4ce6-845e-50077f332e29" (UID: "821784b6-5f58-4ce6-845e-50077f332e29"). InnerVolumeSpecName "kube-api-access-czdlr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:10.723036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.722954    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/821784b6-5f58-4ce6-845e-50077f332e29-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:10.723036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.722983    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-czdlr\" (UniqueName: \"kubernetes.io/projected/821784b6-5f58-4ce6-845e-50077f332e29-kube-api-access-czdlr\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:10.723036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.722995    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:10.723036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.723006    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:10.723036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.723013    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:10.723036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.723022    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w6dvp\" (UniqueName: \"kubernetes.io/projected/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-kube-api-access-w6dvp\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:10.723036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.723030    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:10.723036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.723038    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:10.723036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.723046    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2268303a-b8a6-4783-b4fc-3467cd0c2bb3-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:10.723442 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:10.723053    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/821784b6-5f58-4ce6-845e-50077f332e29-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:11.026950 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:11.026922    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:11.321647 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:11.321558    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"
Nov 02 17:52:11.321647 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:11.321572    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4" event={"ID":"2268303a-b8a6-4783-b4fc-3467cd0c2bb3","Type":"ContainerDied","Data":"e203f4a74424024d2da5a0ce673ce75ed761416ad9269f3e80872b94f388bfe5"}
Nov 02 17:52:11.321647 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:11.321638    4456 scope.go:122] "RemoveContainer" containerID="24ae40e543016679f95f4771c3038102f64f7638708d8e314fe7c78ac01479fc"
Nov 02 17:52:11.323402 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:11.323345    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"
Nov 02 17:52:11.323402 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:11.323361    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p" event={"ID":"821784b6-5f58-4ce6-845e-50077f332e29","Type":"ContainerDied","Data":"a053b7646e243cfaf22e33dab2e8c674f88d56a9a2f2c725545b9bea9e0f4d9b"}
Nov 02 17:52:11.323498 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:11.323411    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a053b7646e243cfaf22e33dab2e8c674f88d56a9a2f2c725545b9bea9e0f4d9b"
Nov 02 17:52:11.328993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:11.328965    4456 generic.go:356] "Generic (PLEG): container finished" podID="6016b480-9073-45a6-b00b-84d4ab57d67b" containerID="5241d39e91ac4d84aff35eac60f4c39c00b61c2b5bf77e9958329cf519f18cbc" exitCode=0
Nov 02 17:52:11.329171 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:11.329132    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh" event={"ID":"6016b480-9073-45a6-b00b-84d4ab57d67b","Type":"ContainerDied","Data":"5241d39e91ac4d84aff35eac60f4c39c00b61c2b5bf77e9958329cf519f18cbc"}
Nov 02 17:52:11.394640 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:11.394608    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"]
Nov 02 17:52:11.405681 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:11.405649    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-0d1f5d84-openshift-default-68fdfd9cd9-rxmb4"]
Nov 02 17:52:11.621695 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:11.621610    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" path="/var/lib/kubelet/pods/2268303a-b8a6-4783-b4fc-3467cd0c2bb3/volumes"
Nov 02 17:52:12.527780 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.527748    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"
Nov 02 17:52:12.661305 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.661227    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6016b480-9073-45a6-b00b-84d4ab57d67b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6016b480-9073-45a6-b00b-84d4ab57d67b-maas-api-xks-ca\") pod \"6016b480-9073-45a6-b00b-84d4ab57d67b\" (UID: \"6016b480-9073-45a6-b00b-84d4ab57d67b\") "
Nov 02 17:52:12.661537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.661517    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6016b480-9073-45a6-b00b-84d4ab57d67b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6016b480-9073-45a6-b00b-84d4ab57d67b-maas-api-openshift-ca\") pod \"6016b480-9073-45a6-b00b-84d4ab57d67b\" (UID: \"6016b480-9073-45a6-b00b-84d4ab57d67b\") "
Nov 02 17:52:12.661676 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.661660    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6016b480-9073-45a6-b00b-84d4ab57d67b-kube-api-access-gg767\" (UniqueName: \"kubernetes.io/projected/6016b480-9073-45a6-b00b-84d4ab57d67b-kube-api-access-gg767\") pod \"6016b480-9073-45a6-b00b-84d4ab57d67b\" (UID: \"6016b480-9073-45a6-b00b-84d4ab57d67b\") "
Nov 02 17:52:12.661948 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.661922    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6016b480-9073-45a6-b00b-84d4ab57d67b-maas-api-openshift-ca" pod "6016b480-9073-45a6-b00b-84d4ab57d67b" (UID: "6016b480-9073-45a6-b00b-84d4ab57d67b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:12.662361 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.662337    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6016b480-9073-45a6-b00b-84d4ab57d67b-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:12.663634 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.663604    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6016b480-9073-45a6-b00b-84d4ab57d67b-maas-api-xks-ca" pod "6016b480-9073-45a6-b00b-84d4ab57d67b" (UID: "6016b480-9073-45a6-b00b-84d4ab57d67b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:52:12.663990 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.663968    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6016b480-9073-45a6-b00b-84d4ab57d67b-kube-api-access-gg767" pod "6016b480-9073-45a6-b00b-84d4ab57d67b" (UID: "6016b480-9073-45a6-b00b-84d4ab57d67b"). InnerVolumeSpecName "kube-api-access-gg767". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:12.730274 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.730240    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"]
Nov 02 17:52:12.731458 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.731433    4456 cpu_manager.go:414] "containerMap: removing container" podUID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerName="istio-proxy"
Nov 02 17:52:12.731458 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.731458    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerName="istio-proxy"
Nov 02 17:52:12.731652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.731502    4456 cpu_manager.go:414] "containerMap: removing container" podUID="821784b6-5f58-4ce6-845e-50077f332e29" containerName="revoke-keys"
Nov 02 17:52:12.731652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.731512    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="821784b6-5f58-4ce6-845e-50077f332e29" containerName="revoke-keys"
Nov 02 17:52:12.731652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.731545    4456 cpu_manager.go:414] "containerMap: removing container" podUID="6016b480-9073-45a6-b00b-84d4ab57d67b" containerName="revoke-keys"
Nov 02 17:52:12.731652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.731553    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6016b480-9073-45a6-b00b-84d4ab57d67b" containerName="revoke-keys"
Nov 02 17:52:12.731652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.731573    4456 cpu_manager.go:414] "containerMap: removing container" podUID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerName="istio-proxy"
Nov 02 17:52:12.731652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.731581    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerName="istio-proxy"
Nov 02 17:52:12.731976 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.731762    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerName="istio-proxy"
Nov 02 17:52:12.731976 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.731780    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="821784b6-5f58-4ce6-845e-50077f332e29" containerName="revoke-keys"
Nov 02 17:52:12.731976 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.731792    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6016b480-9073-45a6-b00b-84d4ab57d67b" containerName="revoke-keys"
Nov 02 17:52:12.733668 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.733639    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:12.736404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.736385    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:12.758604 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.758580    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-inf-a-063f48-serving-cert\""
Nov 02 17:52:12.763655 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.763621    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"]
Nov 02 17:52:12.764509 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.764480    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gg767\" (UniqueName: \"kubernetes.io/projected/6016b480-9073-45a6-b00b-84d4ab57d67b-kube-api-access-gg767\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:12.764596 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.764515    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6016b480-9073-45a6-b00b-84d4ab57d67b-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:12.866425 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.866389    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/712733fd-b069-40c2-b82c-7a86005841c0-maas-api-tls\") pod \"maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:12.866663 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.866637    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712733fd-b069-40c2-b82c-7a86005841c0-metrics-certs\") pod \"maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:12.866710 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.866695    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/712733fd-b069-40c2-b82c-7a86005841c0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:12.866891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.866875    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q76q\" (UniqueName: \"kubernetes.io/projected/712733fd-b069-40c2-b82c-7a86005841c0-kube-api-access-2q76q\") pod \"maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:12.968363 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.968324    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:12.968949 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.968915    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712733fd-b069-40c2-b82c-7a86005841c0-metrics-certs\") pod \"maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:12.969116 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.968986    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/712733fd-b069-40c2-b82c-7a86005841c0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:12.969116 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.969104    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q76q\" (UniqueName: \"kubernetes.io/projected/712733fd-b069-40c2-b82c-7a86005841c0-kube-api-access-2q76q\") pod \"maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:12.969327 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.969220    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/712733fd-b069-40c2-b82c-7a86005841c0-maas-api-tls\") pod \"maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:12.973619 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.971663    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/712733fd-b069-40c2-b82c-7a86005841c0-openshift-service-ca\") pod \"maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:12.998928 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:12.996504    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/712733fd-b069-40c2-b82c-7a86005841c0-maas-api-tls\") pod \"maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:13.009837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:13.000410    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712733fd-b069-40c2-b82c-7a86005841c0-metrics-certs\") pod \"maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:13.022962 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:13.022919    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q76q\" (UniqueName: \"kubernetes.io/projected/712733fd-b069-40c2-b82c-7a86005841c0-kube-api-access-2q76q\") pod \"maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") " pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:13.058230 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:13.058194    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:13.141129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:13.141049    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:13.348860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:13.348751    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"
Nov 02 17:52:13.348860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:13.348785    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh" event={"ID":"6016b480-9073-45a6-b00b-84d4ab57d67b","Type":"ContainerDied","Data":"d28d7ff8a800f88bdc7201130b065251cce8c97f4c4e61047376a1de2c9e295b"}
Nov 02 17:52:13.349042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:13.348869    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d28d7ff8a800f88bdc7201130b065251cce8c97f4c4e61047376a1de2c9e295b"
Nov 02 17:52:13.392583 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:13.392545    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"]
Nov 02 17:52:13.393260 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:52:13.393229    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod712733fd_b069_40c2_b82c_7a86005841c0.slice/crio-84c56a4b01e8a518c60a2977b79b8f1da0dc24a7e631218a0bf9c99fc4d4e60e WatchSource:0}: Error finding container 84c56a4b01e8a518c60a2977b79b8f1da0dc24a7e631218a0bf9c99fc4d4e60e: Status 404 returned error can't find the container with id 84c56a4b01e8a518c60a2977b79b8f1da0dc24a7e631218a0bf9c99fc4d4e60e
Nov 02 17:52:14.367541 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:14.365284    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4" event={"ID":"712733fd-b069-40c2-b82c-7a86005841c0","Type":"ContainerStarted","Data":"928e46cb358a206070aead24b8b5fce080e7a3ef9e1fb8e3d40a9f31b8e52374"}
Nov 02 17:52:14.367541 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:14.366305    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4" event={"ID":"712733fd-b069-40c2-b82c-7a86005841c0","Type":"ContainerStarted","Data":"84c56a4b01e8a518c60a2977b79b8f1da0dc24a7e631218a0bf9c99fc4d4e60e"}
Nov 02 17:52:14.367541 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:14.366376    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:14.414555 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:14.414507    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4" podStartSLOduration=1.934407662 podStartE2EDuration="2.414490847s" totalImagesPullingTime="480.083185ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:14.411996951 +0000 UTC m=+1350.903060113" watchObservedRunningTime="2026-11-02 17:52:14.414490847 +0000 UTC m=+1350.905554008"
Nov 02 17:52:14.777363 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:14.777210    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:15.037850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:15.037746    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"]
Nov 02 17:52:15.049908 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:15.049875    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-94e3ead7-bhrxm"]
Nov 02 17:52:15.620600 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:15.620563    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="51884bda-c8ca-483a-b9da-f85e5624c1c2" path="/var/lib/kubelet/pods/51884bda-c8ca-483a-b9da-f85e5624c1c2/volumes"
Nov 02 17:52:16.168016 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:52:16.167978    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:52:16.686403 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:16.686360    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:17.871561 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:17.871523    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"]
Nov 02 17:52:17.909324 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:17.909284    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"]
Nov 02 17:52:18.051744 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:18.051706    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:18.197869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:18.197836    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:18.431512 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:18.431468    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:19.656768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:19.656727    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:19.920445 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:19.920352    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:20.179803 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:20.179670    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:20.379533 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:20.379502    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 17:52:21.899717 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:21.899685    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:22.872176 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:22.872108    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d" podUID="c24df51d-d414-475e-8850-99604ace85dc" containerName="payload-processing" containerID="cri-o://7fe40d10ca68b3279d6e983abecbe433687aaf4f3934af746a67014a9263daa6" gracePeriod=25
Nov 02 17:52:22.909872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:22.909830    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f" podUID="ee1be5c9-f0b9-48b9-a396-540b41d57247" containerName="payload-pre-processing" containerID="cri-o://2a33e195aadc2cd8b8c7ef4662a0674fbd2ccc0cfc3095d4c12599f1e17bc12c" gracePeriod=25
Nov 02 17:52:23.231569 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.231542    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"
Nov 02 17:52:23.295242 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.295208    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"
Nov 02 17:52:23.380713 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.380675    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ee1be5c9-f0b9-48b9-a396-540b41d57247-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee1be5c9-f0b9-48b9-a396-540b41d57247-plugins-config-volume\") pod \"ee1be5c9-f0b9-48b9-a396-540b41d57247\" (UID: \"ee1be5c9-f0b9-48b9-a396-540b41d57247\") "
Nov 02 17:52:23.380961 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.380761    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c24df51d-d414-475e-8850-99604ace85dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24df51d-d414-475e-8850-99604ace85dc-plugins-config-volume\") pod \"c24df51d-d414-475e-8850-99604ace85dc\" (UID: \"c24df51d-d414-475e-8850-99604ace85dc\") "
Nov 02 17:52:23.381040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.380971    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ee1be5c9-f0b9-48b9-a396-540b41d57247-kube-api-access-g2rpd\" (UniqueName: \"kubernetes.io/projected/ee1be5c9-f0b9-48b9-a396-540b41d57247-kube-api-access-g2rpd\") pod \"ee1be5c9-f0b9-48b9-a396-540b41d57247\" (UID: \"ee1be5c9-f0b9-48b9-a396-540b41d57247\") "
Nov 02 17:52:23.381526 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.381220    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c24df51d-d414-475e-8850-99604ace85dc-kube-api-access-hq274\" (UniqueName: \"kubernetes.io/projected/c24df51d-d414-475e-8850-99604ace85dc-kube-api-access-hq274\") pod \"c24df51d-d414-475e-8850-99604ace85dc\" (UID: \"c24df51d-d414-475e-8850-99604ace85dc\") "
Nov 02 17:52:23.381850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.381799    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c24df51d-d414-475e-8850-99604ace85dc-plugins-config-volume" pod "c24df51d-d414-475e-8850-99604ace85dc" (UID: "c24df51d-d414-475e-8850-99604ace85dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:23.382237 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.382071    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ee1be5c9-f0b9-48b9-a396-540b41d57247-plugins-config-volume" pod "ee1be5c9-f0b9-48b9-a396-540b41d57247" (UID: "ee1be5c9-f0b9-48b9-a396-540b41d57247"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:23.382237 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.382200    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/ee1be5c9-f0b9-48b9-a396-540b41d57247-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:23.382237 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.382217    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c24df51d-d414-475e-8850-99604ace85dc-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:23.384678 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.384586    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c24df51d-d414-475e-8850-99604ace85dc-kube-api-access-hq274" pod "c24df51d-d414-475e-8850-99604ace85dc" (UID: "c24df51d-d414-475e-8850-99604ace85dc"). InnerVolumeSpecName "kube-api-access-hq274". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:23.384678 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.384642    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ee1be5c9-f0b9-48b9-a396-540b41d57247-kube-api-access-g2rpd" pod "ee1be5c9-f0b9-48b9-a396-540b41d57247" (UID: "ee1be5c9-f0b9-48b9-a396-540b41d57247"). InnerVolumeSpecName "kube-api-access-g2rpd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:23.451580 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.451552    4456 generic.go:356] "Generic (PLEG): container finished" podID="c24df51d-d414-475e-8850-99604ace85dc" containerID="7fe40d10ca68b3279d6e983abecbe433687aaf4f3934af746a67014a9263daa6" exitCode=0
Nov 02 17:52:23.451755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.451599    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"
Nov 02 17:52:23.451755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.451642    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d" event={"ID":"c24df51d-d414-475e-8850-99604ace85dc","Type":"ContainerDied","Data":"7fe40d10ca68b3279d6e983abecbe433687aaf4f3934af746a67014a9263daa6"}
Nov 02 17:52:23.451755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.451691    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d" event={"ID":"c24df51d-d414-475e-8850-99604ace85dc","Type":"ContainerDied","Data":"3f95519553b191e2ad95fa866036cbc5594537f5ecce188acc5f598d7cb49a34"}
Nov 02 17:52:23.451755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.451723    4456 scope.go:122] "RemoveContainer" containerID="7fe40d10ca68b3279d6e983abecbe433687aaf4f3934af746a67014a9263daa6"
Nov 02 17:52:23.453297 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.453273    4456 generic.go:356] "Generic (PLEG): container finished" podID="ee1be5c9-f0b9-48b9-a396-540b41d57247" containerID="2a33e195aadc2cd8b8c7ef4662a0674fbd2ccc0cfc3095d4c12599f1e17bc12c" exitCode=0
Nov 02 17:52:23.453405 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.453303    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f" event={"ID":"ee1be5c9-f0b9-48b9-a396-540b41d57247","Type":"ContainerDied","Data":"2a33e195aadc2cd8b8c7ef4662a0674fbd2ccc0cfc3095d4c12599f1e17bc12c"}
Nov 02 17:52:23.453405 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.453318    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"
Nov 02 17:52:23.453405 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.453352    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f" event={"ID":"ee1be5c9-f0b9-48b9-a396-540b41d57247","Type":"ContainerDied","Data":"7ab94e88683e7f62bd9e3d4d02e5c6aaaa89c046e821d9d2fb32fc290161335e"}
Nov 02 17:52:23.463495 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.463472    4456 scope.go:122] "RemoveContainer" containerID="7fe40d10ca68b3279d6e983abecbe433687aaf4f3934af746a67014a9263daa6"
Nov 02 17:52:23.463767 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:52:23.463747    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7fe40d10ca68b3279d6e983abecbe433687aaf4f3934af746a67014a9263daa6\": container with ID starting with 7fe40d10ca68b3279d6e983abecbe433687aaf4f3934af746a67014a9263daa6 not found: ID does not exist" containerID="7fe40d10ca68b3279d6e983abecbe433687aaf4f3934af746a67014a9263daa6"
Nov 02 17:52:23.463879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.463774    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7fe40d10ca68b3279d6e983abecbe433687aaf4f3934af746a67014a9263daa6"} err="failed to get container status \"7fe40d10ca68b3279d6e983abecbe433687aaf4f3934af746a67014a9263daa6\": rpc error: code = NotFound desc = could not find container \"7fe40d10ca68b3279d6e983abecbe433687aaf4f3934af746a67014a9263daa6\": container with ID starting with 7fe40d10ca68b3279d6e983abecbe433687aaf4f3934af746a67014a9263daa6 not found: ID does not exist"
Nov 02 17:52:23.463879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.463791    4456 scope.go:122] "RemoveContainer" containerID="2a33e195aadc2cd8b8c7ef4662a0674fbd2ccc0cfc3095d4c12599f1e17bc12c"
Nov 02 17:52:23.474498 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.474475    4456 scope.go:122] "RemoveContainer" containerID="2a33e195aadc2cd8b8c7ef4662a0674fbd2ccc0cfc3095d4c12599f1e17bc12c"
Nov 02 17:52:23.474850 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:52:23.474788    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"2a33e195aadc2cd8b8c7ef4662a0674fbd2ccc0cfc3095d4c12599f1e17bc12c\": container with ID starting with 2a33e195aadc2cd8b8c7ef4662a0674fbd2ccc0cfc3095d4c12599f1e17bc12c not found: ID does not exist" containerID="2a33e195aadc2cd8b8c7ef4662a0674fbd2ccc0cfc3095d4c12599f1e17bc12c"
Nov 02 17:52:23.476860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.476788    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"2a33e195aadc2cd8b8c7ef4662a0674fbd2ccc0cfc3095d4c12599f1e17bc12c"} err="failed to get container status \"2a33e195aadc2cd8b8c7ef4662a0674fbd2ccc0cfc3095d4c12599f1e17bc12c\": rpc error: code = NotFound desc = could not find container \"2a33e195aadc2cd8b8c7ef4662a0674fbd2ccc0cfc3095d4c12599f1e17bc12c\": container with ID starting with 2a33e195aadc2cd8b8c7ef4662a0674fbd2ccc0cfc3095d4c12599f1e17bc12c not found: ID does not exist"
Nov 02 17:52:23.483713 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.483688    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g2rpd\" (UniqueName: \"kubernetes.io/projected/ee1be5c9-f0b9-48b9-a396-540b41d57247-kube-api-access-g2rpd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:23.483802 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.483719    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hq274\" (UniqueName: \"kubernetes.io/projected/c24df51d-d414-475e-8850-99604ace85dc-kube-api-access-hq274\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:23.515586 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.515557    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"]
Nov 02 17:52:23.521959 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.521936    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-f99b8b-6dcf8b7f5-rzs8f"]
Nov 02 17:52:23.556294 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.556268    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"]
Nov 02 17:52:23.566304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.566252    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-f99b8b-7b5c8c97b7-q7p2d"]
Nov 02 17:52:23.620668 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.620631    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c24df51d-d414-475e-8850-99604ace85dc" path="/var/lib/kubelet/pods/c24df51d-d414-475e-8850-99604ace85dc/volumes"
Nov 02 17:52:23.621334 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.621314    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ee1be5c9-f0b9-48b9-a396-540b41d57247" path="/var/lib/kubelet/pods/ee1be5c9-f0b9-48b9-a396-540b41d57247/volumes"
Nov 02 17:52:23.687791 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:23.687584    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:25.443501 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:25.443455    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:27.215858 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:27.215773    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:27.405593 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:27.405557    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:30.170945 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:30.170886    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:30.181217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:30.181183    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:52:30.270893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:30.270583    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded"
Nov 02 17:52:30.292956 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:30.292909    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:52:30.344031 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:30.343973    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:52:30.375323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:30.375272    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:52:30.459370 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:30.459286    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:30.581649 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:30.581038    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:52:30.668884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:30.663909    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:52:30.726907 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:30.722906    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:52:30.787941 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:30.787898    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:52:30.823842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:30.799353    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:52:30.921841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:30.920905    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:52:31.197843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.185892    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 17:52:31.197843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.186512    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:52:31.209841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.207995    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vrvf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:52:31.209841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.208076    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:52:31.377771 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.377727    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:52:31.378527 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.378507    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:52:31.475534 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.469174    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 17:52:31.530140 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.530109    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 17:52:31.641147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.641116    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"]
Nov 02 17:52:31.641879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.641857    4456 cpu_manager.go:414] "containerMap: removing container" podUID="ee1be5c9-f0b9-48b9-a396-540b41d57247" containerName="payload-pre-processing"
Nov 02 17:52:31.641879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.641874    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ee1be5c9-f0b9-48b9-a396-540b41d57247" containerName="payload-pre-processing"
Nov 02 17:52:31.642078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.641889    4456 cpu_manager.go:414] "containerMap: removing container" podUID="c24df51d-d414-475e-8850-99604ace85dc" containerName="payload-processing"
Nov 02 17:52:31.642078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.641894    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c24df51d-d414-475e-8850-99604ace85dc" containerName="payload-processing"
Nov 02 17:52:31.642078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.642046    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ee1be5c9-f0b9-48b9-a396-540b41d57247" containerName="payload-pre-processing"
Nov 02 17:52:31.642078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.642055    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="c24df51d-d414-475e-8850-99604ace85dc" containerName="payload-processing"
Nov 02 17:52:31.642078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.642064    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="2268303a-b8a6-4783-b4fc-3467cd0c2bb3" containerName="istio-proxy"
Nov 02 17:52:31.651605 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.649632    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"
Nov 02 17:52:31.669021 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.663660    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"]
Nov 02 17:52:31.669021 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.667871    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:52:31.669021 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.668568    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:52:31.767234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.767069    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zdtt5\" (UniqueName: \"kubernetes.io/projected/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-kube-api-access-zdtt5\") pod \"maas-api-revoke-keys-e2e-api-a-786c42-6tcn8\" (UID: \"d045ce96-3b4a-4f2b-b928-a30bd681dcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"
Nov 02 17:52:31.767436 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.767263    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-786c42-6tcn8\" (UID: \"d045ce96-3b4a-4f2b-b928-a30bd681dcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"
Nov 02 17:52:31.767570 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.767539    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-786c42-6tcn8\" (UID: \"d045ce96-3b4a-4f2b-b928-a30bd681dcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"
Nov 02 17:52:31.869661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.869622    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zdtt5\" (UniqueName: \"kubernetes.io/projected/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-kube-api-access-zdtt5\") pod \"maas-api-revoke-keys-e2e-api-a-786c42-6tcn8\" (UID: \"d045ce96-3b4a-4f2b-b928-a30bd681dcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"
Nov 02 17:52:31.869920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.869851    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-786c42-6tcn8\" (UID: \"d045ce96-3b4a-4f2b-b928-a30bd681dcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"
Nov 02 17:52:31.870008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.869991    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-786c42-6tcn8\" (UID: \"d045ce96-3b4a-4f2b-b928-a30bd681dcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"
Nov 02 17:52:31.870738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.870666    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-786c42-6tcn8\" (UID: \"d045ce96-3b4a-4f2b-b928-a30bd681dcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"
Nov 02 17:52:31.872597 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.872569    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-786c42-6tcn8\" (UID: \"d045ce96-3b4a-4f2b-b928-a30bd681dcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"
Nov 02 17:52:31.919348 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:31.919308    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zdtt5\" (UniqueName: \"kubernetes.io/projected/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-kube-api-access-zdtt5\") pod \"maas-api-revoke-keys-e2e-api-a-786c42-6tcn8\" (UID: \"d045ce96-3b4a-4f2b-b928-a30bd681dcb3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"
Nov 02 17:52:32.025735 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:32.025630    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"
Nov 02 17:52:32.230077 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:32.230038    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:32.375981 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:32.375893    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"]
Nov 02 17:52:33.050050 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:33.049998    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8" event={"ID":"d045ce96-3b4a-4f2b-b928-a30bd681dcb3","Type":"ContainerStarted","Data":"53f3717cd9169cd86fece72c8d5ee851d06841fac83a6747026785374941a223"}
Nov 02 17:52:33.129958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:33.129740    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:33.221211 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:33.221175    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"]
Nov 02 17:52:33.229830 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:33.229790    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-82fwt"]
Nov 02 17:52:33.274478 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:33.274444    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:33.422903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:33.422167    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:33.643598 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:33.643569    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0bbc0b5-0c76-4471-950b-b07db0a89542" path="/var/lib/kubelet/pods/c0bbc0b5-0c76-4471-950b-b07db0a89542/volumes"
Nov 02 17:52:34.059755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:34.059664    4456 generic.go:356] "Generic (PLEG): container finished" podID="d045ce96-3b4a-4f2b-b928-a30bd681dcb3" containerID="72949d142729cd47f40c5b13b9d137f93f6fb37ba572784be324dc2b0d21ebab" exitCode=0
Nov 02 17:52:34.059960 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:34.059745    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8" event={"ID":"d045ce96-3b4a-4f2b-b928-a30bd681dcb3","Type":"ContainerDied","Data":"72949d142729cd47f40c5b13b9d137f93f6fb37ba572784be324dc2b0d21ebab"}
Nov 02 17:52:35.142220 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.142186    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:35.273067 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.273040    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"
Nov 02 17:52:35.287470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.287426    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"]
Nov 02 17:52:35.288269 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.288247    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d045ce96-3b4a-4f2b-b928-a30bd681dcb3" containerName="revoke-keys"
Nov 02 17:52:35.288269 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.288267    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d045ce96-3b4a-4f2b-b928-a30bd681dcb3" containerName="revoke-keys"
Nov 02 17:52:35.288445 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.288430    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d045ce96-3b4a-4f2b-b928-a30bd681dcb3" containerName="revoke-keys"
Nov 02 17:52:35.292087 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.292067    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:35.310491 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.310448    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-api-a-c3c112-serving-cert\""
Nov 02 17:52:35.319059 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.319029    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"]
Nov 02 17:52:35.351285 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.351194    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-maas-api-xks-ca\") pod \"d045ce96-3b4a-4f2b-b928-a30bd681dcb3\" (UID: \"d045ce96-3b4a-4f2b-b928-a30bd681dcb3\") "
Nov 02 17:52:35.351285 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.351274    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-kube-api-access-zdtt5\" (UniqueName: \"kubernetes.io/projected/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-kube-api-access-zdtt5\") pod \"d045ce96-3b4a-4f2b-b928-a30bd681dcb3\" (UID: \"d045ce96-3b4a-4f2b-b928-a30bd681dcb3\") "
Nov 02 17:52:35.351521 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.351425    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-maas-api-openshift-ca\") pod \"d045ce96-3b4a-4f2b-b928-a30bd681dcb3\" (UID: \"d045ce96-3b4a-4f2b-b928-a30bd681dcb3\") "
Nov 02 17:52:35.351992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.351944    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-maas-api-openshift-ca" pod "d045ce96-3b4a-4f2b-b928-a30bd681dcb3" (UID: "d045ce96-3b4a-4f2b-b928-a30bd681dcb3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:35.352134 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.352089    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v5bss\" (UniqueName: \"kubernetes.io/projected/8faa6946-48e8-4d72-be2c-080a0c8a83d2-kube-api-access-v5bss\") pod \"maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:35.352588 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.352232    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8faa6946-48e8-4d72-be2c-080a0c8a83d2-metrics-certs\") pod \"maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:35.352588 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.352502    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8faa6946-48e8-4d72-be2c-080a0c8a83d2-maas-api-tls\") pod \"maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:35.352588 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.352542    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8faa6946-48e8-4d72-be2c-080a0c8a83d2-openshift-service-ca\") pod \"maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:35.353347 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.353319    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:35.354402 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.354372    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-kube-api-access-zdtt5" pod "d045ce96-3b4a-4f2b-b928-a30bd681dcb3" (UID: "d045ce96-3b4a-4f2b-b928-a30bd681dcb3"). InnerVolumeSpecName "kube-api-access-zdtt5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:35.354543 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.354521    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-maas-api-xks-ca" pod "d045ce96-3b4a-4f2b-b928-a30bd681dcb3" (UID: "d045ce96-3b4a-4f2b-b928-a30bd681dcb3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:52:35.423847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.423797    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:35.455055 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.455024    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v5bss\" (UniqueName: \"kubernetes.io/projected/8faa6946-48e8-4d72-be2c-080a0c8a83d2-kube-api-access-v5bss\") pod \"maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:35.455217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.455060    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8faa6946-48e8-4d72-be2c-080a0c8a83d2-metrics-certs\") pod \"maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:35.455333 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.455242    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8faa6946-48e8-4d72-be2c-080a0c8a83d2-maas-api-tls\") pod \"maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:35.455333 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.455270    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8faa6946-48e8-4d72-be2c-080a0c8a83d2-openshift-service-ca\") pod \"maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:35.455517 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.455499    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:35.455584 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.455518    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zdtt5\" (UniqueName: \"kubernetes.io/projected/d045ce96-3b4a-4f2b-b928-a30bd681dcb3-kube-api-access-zdtt5\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:35.456103 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.456074    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8faa6946-48e8-4d72-be2c-080a0c8a83d2-openshift-service-ca\") pod \"maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:35.457642 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.457614    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8faa6946-48e8-4d72-be2c-080a0c8a83d2-metrics-certs\") pod \"maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:35.457766 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.457745    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8faa6946-48e8-4d72-be2c-080a0c8a83d2-maas-api-tls\") pod \"maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:35.478573 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.478545    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v5bss\" (UniqueName: \"kubernetes.io/projected/8faa6946-48e8-4d72-be2c-080a0c8a83d2-kube-api-access-v5bss\") pod \"maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") " pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:35.583881 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.583788    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:35.609111 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.609001    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:35.951764 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:35.951737    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"]
Nov 02 17:52:35.953334 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:52:35.953301    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8faa6946_48e8_4d72_be2c_080a0c8a83d2.slice/crio-f51c962be6a561ffeb48b8790289b01dfcd0255e8d62e5384d72115831e174c2 WatchSource:0}: Error finding container f51c962be6a561ffeb48b8790289b01dfcd0255e8d62e5384d72115831e174c2: Status 404 returned error can't find the container with id f51c962be6a561ffeb48b8790289b01dfcd0255e8d62e5384d72115831e174c2
Nov 02 17:52:36.083975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:36.083942    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"
Nov 02 17:52:36.084137 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:36.083939    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8" event={"ID":"d045ce96-3b4a-4f2b-b928-a30bd681dcb3","Type":"ContainerDied","Data":"53f3717cd9169cd86fece72c8d5ee851d06841fac83a6747026785374941a223"}
Nov 02 17:52:36.084137 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:36.084079    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53f3717cd9169cd86fece72c8d5ee851d06841fac83a6747026785374941a223"
Nov 02 17:52:36.085642 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:36.085614    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc" event={"ID":"8faa6946-48e8-4d72-be2c-080a0c8a83d2","Type":"ContainerStarted","Data":"f51c962be6a561ffeb48b8790289b01dfcd0255e8d62e5384d72115831e174c2"}
Nov 02 17:52:37.098589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:37.098499    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc" event={"ID":"8faa6946-48e8-4d72-be2c-080a0c8a83d2","Type":"ContainerStarted","Data":"5d7b826b2eb6942859781a8002a9eb8d2b374392a32292b13693724ceba3b1fb"}
Nov 02 17:52:37.099103 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:37.098662    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:37.151159 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:37.151092    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc" podStartSLOduration=1.7673553640000002 podStartE2EDuration="2.151070959s" totalImagesPullingTime="383.715595ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:35 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:37.148265926 +0000 UTC m=+1373.639329127" watchObservedRunningTime="2026-11-02 17:52:37.151070959 +0000 UTC m=+1373.642134121"
Nov 02 17:52:37.368199 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:37.367106    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:37.541779 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:37.541731    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:37.895027 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:37.894974    4456 patch_prober.go:29] interesting pod/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:52:37.895207 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:37.895049    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq" podUID="e37acee8-ead8-40d3-8d09-160e87271744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:52:39.186791 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:39.186722    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:40.130243 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:40.130149    4456 generic.go:356] "Generic (PLEG): container finished" podID="e37acee8-ead8-40d3-8d09-160e87271744" containerID="9b2e7216425c06bb756659c307e4b41ecc390921f7e7809ec975eebf56da772c" exitCode=0
Nov 02 17:52:40.130243 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:40.130224    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq" event={"ID":"e37acee8-ead8-40d3-8d09-160e87271744","Type":"ContainerDied","Data":"9b2e7216425c06bb756659c307e4b41ecc390921f7e7809ec975eebf56da772c"}
Nov 02 17:52:40.130774 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:40.130754    4456 scope.go:122] "RemoveContainer" containerID="9b2e7216425c06bb756659c307e4b41ecc390921f7e7809ec975eebf56da772c"
Nov 02 17:52:40.132689 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:40.132650    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:52:40.132775 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:40.132741    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:52:40.132834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:40.132773    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:52:40.738329 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:40.738296    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:40.880614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:40.880565    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:41.031139 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:41.030421    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:41.143171 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:41.143130    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq" event={"ID":"e37acee8-ead8-40d3-8d09-160e87271744","Type":"ContainerStarted","Data":"edc469400bd64078521d131783582abbfa00e459114777a8e5f0ea23e411cb08"}
Nov 02 17:52:41.578400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:41.578369    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:41.643131 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:41.643094    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:52:41.643301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:41.643283    4456 patch_prober.go:29] interesting pod/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 02 17:52:41.643354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:41.643333    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq" podUID="e37acee8-ead8-40d3-8d09-160e87271744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 02 17:52:41.724806 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:41.724768    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"]
Nov 02 17:52:41.729063 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:41.729039    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"
Nov 02 17:52:41.745474 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:41.745450    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-mt-04e682fe\""
Nov 02 17:52:41.746315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:41.746290    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"]
Nov 02 17:52:41.747092 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:41.747069    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:41.833947 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:41.817141    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-mt-04e682fe-dockercfg-7wjxd\""
Nov 02 17:52:41.875785 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:41.862676    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lmwsg\" (UniqueName: \"kubernetes.io/projected/225f3685-fcc5-4593-a4c9-29da1e9c928d-kube-api-access-lmwsg\") pod \"payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7\" (UID: \"225f3685-fcc5-4593-a4c9-29da1e9c928d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"
Nov 02 17:52:41.875785 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:41.862961    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225f3685-fcc5-4593-a4c9-29da1e9c928d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7\" (UID: \"225f3685-fcc5-4593-a4c9-29da1e9c928d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"
Nov 02 17:52:42.130893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:42.126065    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lmwsg\" (UniqueName: \"kubernetes.io/projected/225f3685-fcc5-4593-a4c9-29da1e9c928d-kube-api-access-lmwsg\") pod \"payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7\" (UID: \"225f3685-fcc5-4593-a4c9-29da1e9c928d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"
Nov 02 17:52:42.130893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:42.126153    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225f3685-fcc5-4593-a4c9-29da1e9c928d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7\" (UID: \"225f3685-fcc5-4593-a4c9-29da1e9c928d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"
Nov 02 17:52:42.271366 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:42.271330    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225f3685-fcc5-4593-a4c9-29da1e9c928d-plugins-config-volume\") pod \"payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7\" (UID: \"225f3685-fcc5-4593-a4c9-29da1e9c928d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"
Nov 02 17:52:42.276432 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:42.276402    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"]
Nov 02 17:52:42.278176 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:42.278119    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lmwsg\" (UniqueName: \"kubernetes.io/projected/225f3685-fcc5-4593-a4c9-29da1e9c928d-kube-api-access-lmwsg\") pod \"payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7\" (UID: \"225f3685-fcc5-4593-a4c9-29da1e9c928d\") " pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"
Nov 02 17:52:42.281839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:42.281789    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"]
Nov 02 17:52:42.345574 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:42.345544    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"
Nov 02 17:52:42.415396 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:42.411843    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:42.572691 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:42.572661    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:42.643802 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:42.643763    4456 patch_prober.go:29] interesting pod/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 02 17:52:42.644015 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:42.643867    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq" podUID="e37acee8-ead8-40d3-8d09-160e87271744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 02 17:52:42.829406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:42.829367    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"]
Nov 02 17:52:43.089548 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:43.089466    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:43.139853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:43.138905    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:52:43.626041 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:43.625994    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:43.643157 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:43.643127    4456 patch_prober.go:29] interesting pod/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 02 17:52:43.643470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:43.643441    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq" podUID="e37acee8-ead8-40d3-8d09-160e87271744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 02 17:52:43.648912 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:43.648886    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"
Nov 02 17:52:43.649046 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:43.648942    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7" event={"ID":"225f3685-fcc5-4593-a4c9-29da1e9c928d","Type":"ContainerStarted","Data":"a70b4f24a2d3943b27a5b5c248070d3b07a229faf1b652ddd4e244c25a62e3d0"}
Nov 02 17:52:43.649046 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:43.648972    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7" event={"ID":"225f3685-fcc5-4593-a4c9-29da1e9c928d","Type":"ContainerStarted","Data":"246da8bfa7288a718dd41eefbce77d391cfcdb15b3a7d925b0f3450a73d3d36a"}
Nov 02 17:52:43.716550 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:43.716488    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7" podStartSLOduration=2.716467497 podStartE2EDuration="2.716467497s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:52:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:52:43.694543932 +0000 UTC m=+1380.185607094" watchObservedRunningTime="2026-11-02 17:52:43.716467497 +0000 UTC m=+1380.207530661"
Nov 02 17:52:43.773438 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:43.773383    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:44.643248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:44.643209    4456 patch_prober.go:29] interesting pod/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 02 17:52:44.643840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:44.643792    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq" podUID="e37acee8-ead8-40d3-8d09-160e87271744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 02 17:52:45.643560 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:45.643517    4456 patch_prober.go:29] interesting pod/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 02 17:52:45.644096 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:45.643571    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq" podUID="e37acee8-ead8-40d3-8d09-160e87271744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 02 17:52:45.977989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:45.977954    4456 scope.go:122] "RemoveContainer" containerID="6fd13213f39a7920736e9fee8d27e3fc7530e7a22bde03f78e52c8b4a2141052"
Nov 02 17:52:46.024421 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:46.024400    4456 scope.go:122] "RemoveContainer" containerID="09d2b13134cdabf9f47a6d7f1eba5e151f3dddf0d0303477f9cb35987e7f49b5"
Nov 02 17:52:46.171473 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:52:46.171441    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:52:46.638330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:46.638258    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:46.643199 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:46.643175    4456 patch_prober.go:29] interesting pod/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused" start-of-body=
Nov 02 17:52:46.643310 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:46.643220    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq" podUID="e37acee8-ead8-40d3-8d09-160e87271744" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.147:15021/healthz/ready\": dial tcp 10.134.0.147:15021: connect: connection refused"
Nov 02 17:52:46.782887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:46.782794    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:47.277929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.277868    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp" podUID="d21bb8e7-fe0a-4654-8373-429d9a092116" containerName="payload-processing" containerID="cri-o://c6d7b6ab73f2b06d67ed3aff51f34f02e7c80933c15c7c07b1676c73dde51a26" gracePeriod=25
Nov 02 17:52:47.282448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.282411    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n" podUID="4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0" containerName="payload-pre-processing" containerID="cri-o://9e18613b05c3c7712c524b41dc4770d93aa259fdf33c2b37ca3a8bbeb9111fda" gracePeriod=25
Nov 02 17:52:47.618859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.618614    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"
Nov 02 17:52:47.649318 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.649292    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:52:47.649723 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.649678    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:52:47.653643 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.653624    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:52:47.674854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.674742    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d21bb8e7-fe0a-4654-8373-429d9a092116-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d21bb8e7-fe0a-4654-8373-429d9a092116-plugins-config-volume\") pod \"d21bb8e7-fe0a-4654-8373-429d9a092116\" (UID: \"d21bb8e7-fe0a-4654-8373-429d9a092116\") "
Nov 02 17:52:47.675314 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.675162    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d21bb8e7-fe0a-4654-8373-429d9a092116-plugins-config-volume" pod "d21bb8e7-fe0a-4654-8373-429d9a092116" (UID: "d21bb8e7-fe0a-4654-8373-429d9a092116"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:47.676489 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.675582    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d21bb8e7-fe0a-4654-8373-429d9a092116-kube-api-access-wkj5w\" (UniqueName: \"kubernetes.io/projected/d21bb8e7-fe0a-4654-8373-429d9a092116-kube-api-access-wkj5w\") pod \"d21bb8e7-fe0a-4654-8373-429d9a092116\" (UID: \"d21bb8e7-fe0a-4654-8373-429d9a092116\") "
Nov 02 17:52:47.676839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.676636    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"
Nov 02 17:52:47.676839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.676793    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d21bb8e7-fe0a-4654-8373-429d9a092116-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:47.679419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.679398    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d21bb8e7-fe0a-4654-8373-429d9a092116-kube-api-access-wkj5w" pod "d21bb8e7-fe0a-4654-8373-429d9a092116" (UID: "d21bb8e7-fe0a-4654-8373-429d9a092116"). InnerVolumeSpecName "kube-api-access-wkj5w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:47.726357 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.717535    4456 generic.go:356] "Generic (PLEG): container finished" podID="d21bb8e7-fe0a-4654-8373-429d9a092116" containerID="c6d7b6ab73f2b06d67ed3aff51f34f02e7c80933c15c7c07b1676c73dde51a26" exitCode=0
Nov 02 17:52:47.726357 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.717689    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"
Nov 02 17:52:47.726357 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.721381    4456 generic.go:356] "Generic (PLEG): container finished" podID="4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0" containerID="9e18613b05c3c7712c524b41dc4770d93aa259fdf33c2b37ca3a8bbeb9111fda" exitCode=0
Nov 02 17:52:47.726357 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.722421    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"
Nov 02 17:52:47.726357 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.722672    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp" event={"ID":"d21bb8e7-fe0a-4654-8373-429d9a092116","Type":"ContainerDied","Data":"c6d7b6ab73f2b06d67ed3aff51f34f02e7c80933c15c7c07b1676c73dde51a26"}
Nov 02 17:52:47.726357 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.722725    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp" event={"ID":"d21bb8e7-fe0a-4654-8373-429d9a092116","Type":"ContainerDied","Data":"7db00591c9bf74d2a82a079a17a0ad707da82f46b7327cbdce9a58bc310cbdfd"}
Nov 02 17:52:47.726357 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.722753    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n" event={"ID":"4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0","Type":"ContainerDied","Data":"9e18613b05c3c7712c524b41dc4770d93aa259fdf33c2b37ca3a8bbeb9111fda"}
Nov 02 17:52:47.726357 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.722778    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n" event={"ID":"4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0","Type":"ContainerDied","Data":"7a9436c8a3a24bc5320c7a3e4a058fd64f4a8d676700023576750b3f979f3d04"}
Nov 02 17:52:47.726357 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.722806    4456 scope.go:122] "RemoveContainer" containerID="c6d7b6ab73f2b06d67ed3aff51f34f02e7c80933c15c7c07b1676c73dde51a26"
Nov 02 17:52:47.751027 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.750998    4456 scope.go:122] "RemoveContainer" containerID="c6d7b6ab73f2b06d67ed3aff51f34f02e7c80933c15c7c07b1676c73dde51a26"
Nov 02 17:52:47.758984 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:52:47.758952    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c6d7b6ab73f2b06d67ed3aff51f34f02e7c80933c15c7c07b1676c73dde51a26\": container with ID starting with c6d7b6ab73f2b06d67ed3aff51f34f02e7c80933c15c7c07b1676c73dde51a26 not found: ID does not exist" containerID="c6d7b6ab73f2b06d67ed3aff51f34f02e7c80933c15c7c07b1676c73dde51a26"
Nov 02 17:52:47.759122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.758996    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c6d7b6ab73f2b06d67ed3aff51f34f02e7c80933c15c7c07b1676c73dde51a26"} err="failed to get container status \"c6d7b6ab73f2b06d67ed3aff51f34f02e7c80933c15c7c07b1676c73dde51a26\": rpc error: code = NotFound desc = could not find container \"c6d7b6ab73f2b06d67ed3aff51f34f02e7c80933c15c7c07b1676c73dde51a26\": container with ID starting with c6d7b6ab73f2b06d67ed3aff51f34f02e7c80933c15c7c07b1676c73dde51a26 not found: ID does not exist"
Nov 02 17:52:47.759122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.759021    4456 scope.go:122] "RemoveContainer" containerID="9e18613b05c3c7712c524b41dc4770d93aa259fdf33c2b37ca3a8bbeb9111fda"
Nov 02 17:52:47.779900 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.779862    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0-kube-api-access-pwk5p\" (UniqueName: \"kubernetes.io/projected/4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0-kube-api-access-pwk5p\") pod \"4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0\" (UID: \"4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0\") "
Nov 02 17:52:47.785478 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.785446    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0-plugins-config-volume\") pod \"4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0\" (UID: \"4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0\") "
Nov 02 17:52:47.797039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.788941    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0-kube-api-access-pwk5p" pod "4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0" (UID: "4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0"). InnerVolumeSpecName "kube-api-access-pwk5p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:52:47.797039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.790087    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0-plugins-config-volume" pod "4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0" (UID: "4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:52:47.797039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.794780    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wkj5w\" (UniqueName: \"kubernetes.io/projected/d21bb8e7-fe0a-4654-8373-429d9a092116-kube-api-access-wkj5w\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:47.797039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.794903    4456 scope.go:122] "RemoveContainer" containerID="9e18613b05c3c7712c524b41dc4770d93aa259fdf33c2b37ca3a8bbeb9111fda"
Nov 02 17:52:47.797039 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:52:47.795349    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"9e18613b05c3c7712c524b41dc4770d93aa259fdf33c2b37ca3a8bbeb9111fda\": container with ID starting with 9e18613b05c3c7712c524b41dc4770d93aa259fdf33c2b37ca3a8bbeb9111fda not found: ID does not exist" containerID="9e18613b05c3c7712c524b41dc4770d93aa259fdf33c2b37ca3a8bbeb9111fda"
Nov 02 17:52:47.797039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.795385    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"9e18613b05c3c7712c524b41dc4770d93aa259fdf33c2b37ca3a8bbeb9111fda"} err="failed to get container status \"9e18613b05c3c7712c524b41dc4770d93aa259fdf33c2b37ca3a8bbeb9111fda\": rpc error: code = NotFound desc = could not find container \"9e18613b05c3c7712c524b41dc4770d93aa259fdf33c2b37ca3a8bbeb9111fda\": container with ID starting with 9e18613b05c3c7712c524b41dc4770d93aa259fdf33c2b37ca3a8bbeb9111fda not found: ID does not exist"
Nov 02 17:52:47.836643 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.835870    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"]
Nov 02 17:52:47.842874 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.842635    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-a-786c42-7c6f577947-zpthp"]
Nov 02 17:52:47.896395 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.896346    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pwk5p\" (UniqueName: \"kubernetes.io/projected/4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0-kube-api-access-pwk5p\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:47.896395 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:47.896387    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:52:48.085714 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:48.085680    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"]
Nov 02 17:52:48.092282 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:48.092221    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-a-786c42-564fc4c4bc-2qp7n"]
Nov 02 17:52:48.486539 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:48.484872    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:49.617217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:49.617171    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0" path="/var/lib/kubelet/pods/4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0/volumes"
Nov 02 17:52:49.617888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:49.617867    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d21bb8e7-fe0a-4654-8373-429d9a092116" path="/var/lib/kubelet/pods/d21bb8e7-fe0a-4654-8373-429d9a092116/volumes"
Nov 02 17:52:50.365730 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:50.360905    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:52.079974 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:52.077654    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:53.856008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:53.855973    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:54.663653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:54.663621    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"
Nov 02 17:52:55.764061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:55.763973    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:57.556028 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:57.555990    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:59.262238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:59.262198    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:59.500526 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:59.500493    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:52:59.932754 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:52:59.932714    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:00.344076 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.344043    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"]
Nov 02 17:53:00.344856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.344833    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d21bb8e7-fe0a-4654-8373-429d9a092116" containerName="payload-processing"
Nov 02 17:53:00.344856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.344852    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d21bb8e7-fe0a-4654-8373-429d9a092116" containerName="payload-processing"
Nov 02 17:53:00.345036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.344888    4456 cpu_manager.go:414] "containerMap: removing container" podUID="4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0" containerName="payload-pre-processing"
Nov 02 17:53:00.345036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.344893    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0" containerName="payload-pre-processing"
Nov 02 17:53:00.345036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.345033    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="4b8c06ad-f34b-4b8a-9612-f52c5cf0d7e0" containerName="payload-pre-processing"
Nov 02 17:53:00.345135 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.345051    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d21bb8e7-fe0a-4654-8373-429d9a092116" containerName="payload-processing"
Nov 02 17:53:00.350770 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.350745    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.373605 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.373573    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"]
Nov 02 17:53:00.380856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.380789    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b913b2-openshift-default-dockercfg-hrd4n\""
Nov 02 17:53:00.452403 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.452372    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:00.484838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.484794    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gszm5\" (UniqueName: \"kubernetes.io/projected/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-kube-api-access-gszm5\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.485011 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.484933    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-podinfo\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.485060 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.485007    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istiod-ca-cert\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.485113 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.485097    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-credential-socket\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.485223 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.485209    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-data\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.485286 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.485270    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-workload-socket\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.485336 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.485300    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-workload-certs\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.485449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.485435    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-token\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.485492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.485478    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-envoy\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.601891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.597006    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-data\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.601891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.597087    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-workload-socket\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.601891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.597117    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-workload-certs\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.601891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.597227    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-token\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.601891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.597271    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-envoy\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.601891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.597425    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gszm5\" (UniqueName: \"kubernetes.io/projected/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-kube-api-access-gszm5\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.601891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.597461    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-podinfo\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.601891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.597528    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istiod-ca-cert\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.601891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.597599    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-credential-socket\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.603028 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.603000    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-credential-socket\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.603432 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.603412    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-data\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.603780 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.603763    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-workload-socket\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.604338 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.604318    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-workload-certs\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.624396 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.615531    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istiod-ca-cert\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.648837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.636241    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-podinfo\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.650165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.650131    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-envoy\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.671054 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.659700    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:00.688844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.679217    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-token\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.696540 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.696465    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gszm5\" (UniqueName: \"kubernetes.io/projected/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-kube-api-access-gszm5\") pod \"e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") " pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:00.975757 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:00.975729    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:01.160577 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:01.156517    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:01.325795 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:01.325763    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"]
Nov 02 17:53:01.326355 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:53:01.326328    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod65156215_be49_4a1b_a7fd_2b1eb2f3f13d.slice/crio-a123bfea1371bbad94ee368639c886b357d4eb175cf741dd99912d1580687e91 WatchSource:0}: Error finding container a123bfea1371bbad94ee368639c886b357d4eb175cf741dd99912d1580687e91: Status 404 returned error can't find the container with id a123bfea1371bbad94ee368639c886b357d4eb175cf741dd99912d1580687e91
Nov 02 17:53:01.329314 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:01.329281    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:01.329378 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:01.329350    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:01.329419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:01.329380    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:01.879572 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:01.879532    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4" event={"ID":"65156215-be49-4a1b-a7fd-2b1eb2f3f13d","Type":"ContainerStarted","Data":"5d9cae3ca58d585af27bfd3ba6620280816bb0024b13b0622caebfcbac09cf0b"}
Nov 02 17:53:01.880016 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:01.879591    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4" event={"ID":"65156215-be49-4a1b-a7fd-2b1eb2f3f13d","Type":"ContainerStarted","Data":"a123bfea1371bbad94ee368639c886b357d4eb175cf741dd99912d1580687e91"}
Nov 02 17:53:01.940180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:01.940127    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4" podStartSLOduration=1.94011355 podStartE2EDuration="1.94011355s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:01.938087142 +0000 UTC m=+1398.429150333" watchObservedRunningTime="2026-11-02 17:53:01.94011355 +0000 UTC m=+1398.431176711"
Nov 02 17:53:01.976459 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:01.976428    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:02.974845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:02.973316    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:03.003019 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:03.002025    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:03.898187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:03.898144    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:03.899331 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:03.899307    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:53:04.501948 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:04.501909    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:04.652181 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:04.652146    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:04.841590 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:04.841505    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:06.199861 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.199806    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:06.329363 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.329333    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"]
Nov 02 17:53:06.334079 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.334052    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"
Nov 02 17:53:06.351527 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.351490    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:06.351793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.351767    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"]
Nov 02 17:53:06.357416 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.357394    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-b913b2\""
Nov 02 17:53:06.395487 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.395452    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93171ede-4b58-4cdc-9467-f9a6c7574817-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8\" (UID: \"93171ede-4b58-4cdc-9467-f9a6c7574817\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"
Nov 02 17:53:06.395895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.395873    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dd4rk\" (UniqueName: \"kubernetes.io/projected/93171ede-4b58-4cdc-9467-f9a6c7574817-kube-api-access-dd4rk\") pod \"payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8\" (UID: \"93171ede-4b58-4cdc-9467-f9a6c7574817\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"
Nov 02 17:53:06.413318 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.413287    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-b913b2-dockercfg-vwkqh\""
Nov 02 17:53:06.531883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.501154    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93171ede-4b58-4cdc-9467-f9a6c7574817-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8\" (UID: \"93171ede-4b58-4cdc-9467-f9a6c7574817\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"
Nov 02 17:53:06.531883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.501270    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dd4rk\" (UniqueName: \"kubernetes.io/projected/93171ede-4b58-4cdc-9467-f9a6c7574817-kube-api-access-dd4rk\") pod \"payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8\" (UID: \"93171ede-4b58-4cdc-9467-f9a6c7574817\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"
Nov 02 17:53:06.531883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.502620    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93171ede-4b58-4cdc-9467-f9a6c7574817-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8\" (UID: \"93171ede-4b58-4cdc-9467-f9a6c7574817\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"
Nov 02 17:53:06.641937 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.625567    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dd4rk\" (UniqueName: \"kubernetes.io/projected/93171ede-4b58-4cdc-9467-f9a6c7574817-kube-api-access-dd4rk\") pod \"payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8\" (UID: \"93171ede-4b58-4cdc-9467-f9a6c7574817\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"
Nov 02 17:53:06.656984 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.656906    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"
Nov 02 17:53:06.789127 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.770730    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"]
Nov 02 17:53:06.789127 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.777899    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"
Nov 02 17:53:06.789127 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.780298    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"]
Nov 02 17:53:06.859372 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.853200    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l4tcm\" (UniqueName: \"kubernetes.io/projected/7b59811f-df47-421a-9e15-d31e4b7bb248-kube-api-access-l4tcm\") pod \"payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt\" (UID: \"7b59811f-df47-421a-9e15-d31e4b7bb248\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"
Nov 02 17:53:06.860493 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.860463    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b59811f-df47-421a-9e15-d31e4b7bb248-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt\" (UID: \"7b59811f-df47-421a-9e15-d31e4b7bb248\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"
Nov 02 17:53:06.962519 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.962481    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b59811f-df47-421a-9e15-d31e4b7bb248-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt\" (UID: \"7b59811f-df47-421a-9e15-d31e4b7bb248\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"
Nov 02 17:53:06.962720 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.962653    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l4tcm\" (UniqueName: \"kubernetes.io/projected/7b59811f-df47-421a-9e15-d31e4b7bb248-kube-api-access-l4tcm\") pod \"payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt\" (UID: \"7b59811f-df47-421a-9e15-d31e4b7bb248\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"
Nov 02 17:53:06.963248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.963217    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b59811f-df47-421a-9e15-d31e4b7bb248-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt\" (UID: \"7b59811f-df47-421a-9e15-d31e4b7bb248\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"
Nov 02 17:53:06.986640 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:06.986609    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l4tcm\" (UniqueName: \"kubernetes.io/projected/7b59811f-df47-421a-9e15-d31e4b7bb248-kube-api-access-l4tcm\") pod \"payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt\" (UID: \"7b59811f-df47-421a-9e15-d31e4b7bb248\") " pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"
Nov 02 17:53:07.118851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:07.118072    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:07.128908 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:53:07.121563    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod93171ede_4b58_4cdc_9467_f9a6c7574817.slice/crio-48fe516f263c0291ee20c126cba079fd01200ccecea8f7a1fc036af4e5eb8568 WatchSource:0}: Error finding container 48fe516f263c0291ee20c126cba079fd01200ccecea8f7a1fc036af4e5eb8568: Status 404 returned error can't find the container with id 48fe516f263c0291ee20c126cba079fd01200ccecea8f7a1fc036af4e5eb8568
Nov 02 17:53:07.128908 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:07.121665    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"]
Nov 02 17:53:07.174398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:07.174275    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"
Nov 02 17:53:07.497662 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:07.497628    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"]
Nov 02 17:53:07.499002 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:53:07.498970    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7b59811f_df47_421a_9e15_d31e4b7bb248.slice/crio-1c08c37f7f317e0f444b9f9b3c9fa7c2d2443f9cb7a70904fedd5f65a6f659ae WatchSource:0}: Error finding container 1c08c37f7f317e0f444b9f9b3c9fa7c2d2443f9cb7a70904fedd5f65a6f659ae: Status 404 returned error can't find the container with id 1c08c37f7f317e0f444b9f9b3c9fa7c2d2443f9cb7a70904fedd5f65a6f659ae
Nov 02 17:53:07.942993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:07.942952    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt" event={"ID":"7b59811f-df47-421a-9e15-d31e4b7bb248","Type":"ContainerStarted","Data":"dc9a939e2490fa79d04eec61eaf6693e4dc48aa0258af2733a23d95511593245"}
Nov 02 17:53:07.943209 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:07.943020    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt" event={"ID":"7b59811f-df47-421a-9e15-d31e4b7bb248","Type":"ContainerStarted","Data":"1c08c37f7f317e0f444b9f9b3c9fa7c2d2443f9cb7a70904fedd5f65a6f659ae"}
Nov 02 17:53:07.943209 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:07.943049    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"
Nov 02 17:53:07.944674 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:07.944636    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8" event={"ID":"93171ede-4b58-4cdc-9467-f9a6c7574817","Type":"ContainerStarted","Data":"992d2a3e54d4db746448b02f06786266b9c02e56729836a4f100dc885f70cf7f"}
Nov 02 17:53:07.944674 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:07.944678    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8" event={"ID":"93171ede-4b58-4cdc-9467-f9a6c7574817","Type":"ContainerStarted","Data":"48fe516f263c0291ee20c126cba079fd01200ccecea8f7a1fc036af4e5eb8568"}
Nov 02 17:53:07.944917 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:07.944731    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"
Nov 02 17:53:07.984824 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:07.984772    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt" podStartSLOduration=1.984758786 podStartE2EDuration="1.984758786s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:07.983833 +0000 UTC m=+1404.474896168" watchObservedRunningTime="2026-11-02 17:53:07.984758786 +0000 UTC m=+1404.475821947"
Nov 02 17:53:08.025777 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:08.025715    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8" podStartSLOduration=2.025697389 podStartE2EDuration="2.025697389s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:08.024396839 +0000 UTC m=+1404.515460012" watchObservedRunningTime="2026-11-02 17:53:08.025697389 +0000 UTC m=+1404.516760551"
Nov 02 17:53:08.772884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:08.772854    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:10.744756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:10.744709    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:10.911347 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:10.911312    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:12.847958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:12.847902    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:14.476475 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:14.476429    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 17:53:14.685078 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:14.685040    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:14.790017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:14.789911    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:14.987520 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:14.987471    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:15.270964 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:15.270926    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"]
Nov 02 17:53:15.353841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:15.351134    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4f1d499-jt8cp"]
Nov 02 17:53:15.422622 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:15.422566    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:15.457962 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:15.457920    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:15.604903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:15.597754    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:15.680846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:15.657982    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"]
Nov 02 17:53:16.037579 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:16.037528    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0af74da3-80ef-409f-a823-36fbb9b02d85" path="/var/lib/kubelet/pods/0af74da3-80ef-409f-a823-36fbb9b02d85/volumes"
Nov 02 17:53:16.066838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:16.049504    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-f99b8b-6th9p"]
Nov 02 17:53:16.073913 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:16.073881    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:16.266159 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:53:16.265949    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:53:17.475362 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:17.475318    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:17.730666 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:17.730576    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="821784b6-5f58-4ce6-845e-50077f332e29" path="/var/lib/kubelet/pods/821784b6-5f58-4ce6-845e-50077f332e29/volumes"
Nov 02 17:53:18.958491 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:18.958461    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"
Nov 02 17:53:18.959014 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:18.958547    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"
Nov 02 17:53:19.356740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:19.356657    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:21.248694 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.248642    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:21.271445 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.271413    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"]
Nov 02 17:53:21.300555 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.300519    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.305161 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.305129    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"]
Nov 02 17:53:21.404780 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.404749    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.405013 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.404986    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.405065 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.405036    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.405122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.405106    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-n2blw\" (UniqueName: \"kubernetes.io/projected/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-kube-api-access-n2blw\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.405188 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.405174    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.405345 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.405323    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.405404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.405389    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.405458 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.405442    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.405519 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.405504    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.507206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.507121    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.507394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.507228    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.507394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.507285    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.507394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.507316    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.507394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.507369    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.507629 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.507411    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.507629 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.507582    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.507629 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.507614    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.507772 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.507659    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-workload-certs\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.507772 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.507666    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-n2blw\" (UniqueName: \"kubernetes.io/projected/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-kube-api-access-n2blw\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.507772 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.507670    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-data\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.507956 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.507890    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-credential-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.508111 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.508086    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-workload-socket\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.508270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.508249    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istiod-ca-cert\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.510024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.509994    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-envoy\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.510407 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.510387    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-podinfo\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.530927 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.530900    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-token\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.531064 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.531043    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-n2blw\" (UniqueName: \"kubernetes.io/projected/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-kube-api-access-n2blw\") pod \"e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") " pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.642853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.642781    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:21.995424 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:21.995393    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"]
Nov 02 17:53:21.996628 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:53:21.996598    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32d7a3ef_ec4e_4639_b849_baf8078f5fd0.slice/crio-b198d66cd646f7c5b15d91a4a1e67014c5924d12ab65f8f13eb4759997ff10b4 WatchSource:0}: Error finding container b198d66cd646f7c5b15d91a4a1e67014c5924d12ab65f8f13eb4759997ff10b4: Status 404 returned error can't find the container with id b198d66cd646f7c5b15d91a4a1e67014c5924d12ab65f8f13eb4759997ff10b4
Nov 02 17:53:22.000440 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:22.000406    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:22.000562 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:22.000475    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:22.000562 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:22.000503    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:22.183839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:22.180456    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:22.409430 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:22.409397    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql" event={"ID":"32d7a3ef-ec4e-4639-b849-baf8078f5fd0","Type":"ContainerStarted","Data":"d255c53e82a16333f04bcb424be778697aa2e2fbfb8d5e549007236f80fd5375"}
Nov 02 17:53:22.410897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:22.409463    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql" event={"ID":"32d7a3ef-ec4e-4639-b849-baf8078f5fd0","Type":"ContainerStarted","Data":"b198d66cd646f7c5b15d91a4a1e67014c5924d12ab65f8f13eb4759997ff10b4"}
Nov 02 17:53:22.475316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:22.475250    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql" podStartSLOduration=1.4752299739999999 podStartE2EDuration="1.475229974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:22.470016053 +0000 UTC m=+1418.961079238" watchObservedRunningTime="2026-11-02 17:53:22.475229974 +0000 UTC m=+1418.966293138"
Nov 02 17:53:22.645584 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:22.643654    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:22.884841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:22.880481    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:23.027388 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.027349    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:23.264445 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.264406    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:23.268034 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.267069    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"]
Nov 02 17:53:23.268034 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.267242    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7" podUID="98c10ff3-d496-4336-a2cd-921042c719d3" containerName="istio-proxy" containerID="cri-o://77600a881963b6c995f4d2afec219dad9a8788b39af4a147947ed27a989fad54" gracePeriod=30
Nov 02 17:53:23.459887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.459724    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:23.472737 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.467992    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"]
Nov 02 17:53:23.520938 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.517632    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.527183 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.527076    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"]
Nov 02 17:53:23.539518 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.537002    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-api-b-227833-openshift-default-dockercfg-d5n88\""
Nov 02 17:53:23.598840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.594261    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-credential-socket\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.598840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.594372    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-workload-certs\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.598840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.594434    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-t7kh4\" (UniqueName: \"kubernetes.io/projected/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-kube-api-access-t7kh4\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.598840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.594467    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-envoy\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.598840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.594566    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-token\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.598840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.594648    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-workload-socket\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.598840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.594685    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-data\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.598840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.594729    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-podinfo\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.598840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.594803    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istiod-ca-cert\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.680226 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.661034    4456 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 02 17:53:23.680226 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.661103    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql" podUID="32d7a3ef-ec4e-4639-b849-baf8078f5fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 02 17:53:23.680226 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.679925    4456 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 02 17:53:23.680226 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.679983    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp" podUID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 02 17:53:23.710977 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.681774    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:23.895340 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.892874    4456 generic.go:356] "Generic (PLEG): container finished" podID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerID="b53a79d0493a360d98414bc8e408bb9b56548a39e73e6df6cfc4f94d5d63f132" exitCode=0
Nov 02 17:53:23.924324 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.924282    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-token\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.924537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.924408    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-workload-socket\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.924537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.924454    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-data\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.924537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.924503    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-podinfo\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.924737 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.924581    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istiod-ca-cert\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.924905 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.924883    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-credential-socket\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.927982 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.925001    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-workload-certs\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.927982 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.925073    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-t7kh4\" (UniqueName: \"kubernetes.io/projected/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-kube-api-access-t7kh4\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.927982 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.925112    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-envoy\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.927982 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.925767    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istiod-ca-cert\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.927982 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.926062    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-data\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.927982 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.927343    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-workload-socket\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.927982 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.927930    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-credential-socket\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.929085 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.929063    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-workload-certs\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.935759 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.935582    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp" event={"ID":"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e","Type":"ContainerDied","Data":"b53a79d0493a360d98414bc8e408bb9b56548a39e73e6df6cfc4f94d5d63f132"}
Nov 02 17:53:23.936552 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.936532    4456 scope.go:122] "RemoveContainer" containerID="b53a79d0493a360d98414bc8e408bb9b56548a39e73e6df6cfc4f94d5d63f132"
Nov 02 17:53:23.943305 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.943284    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-envoy\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.943758 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.943731    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-podinfo\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.945067 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.945037    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:23.945276 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.945258    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:23.945409 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.945395    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:23.953198 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.950598    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-t7kh4\" (UniqueName: \"kubernetes.io/projected/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-kube-api-access-t7kh4\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:23.953198 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:23.951199    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-token\") pod \"e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") " pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:24.052075 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.050933    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"]
Nov 02 17:53:24.057205 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.057185    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"]
Nov 02 17:53:24.057344 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.057315    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"
Nov 02 17:53:24.074731 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.074706    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:53:24.075012 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.074992    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:53:24.115635 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.115448    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:24.162881 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.141520    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-79zh7\" (UniqueName: \"kubernetes.io/projected/e1d57a75-2039-44db-abae-0085e773803f-kube-api-access-79zh7\") pod \"maas-api-revoke-keys-e2e-api-b-769b70-f4ccj\" (UID: \"e1d57a75-2039-44db-abae-0085e773803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"
Nov 02 17:53:24.164901 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.164870    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d57a75-2039-44db-abae-0085e773803f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-769b70-f4ccj\" (UID: \"e1d57a75-2039-44db-abae-0085e773803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"
Nov 02 17:53:24.165229 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.165209    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d57a75-2039-44db-abae-0085e773803f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-769b70-f4ccj\" (UID: \"e1d57a75-2039-44db-abae-0085e773803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"
Nov 02 17:53:24.205877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.204991    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:24.268156 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.267647    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-79zh7\" (UniqueName: \"kubernetes.io/projected/e1d57a75-2039-44db-abae-0085e773803f-kube-api-access-79zh7\") pod \"maas-api-revoke-keys-e2e-api-b-769b70-f4ccj\" (UID: \"e1d57a75-2039-44db-abae-0085e773803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"
Nov 02 17:53:24.268156 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.267797    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d57a75-2039-44db-abae-0085e773803f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-769b70-f4ccj\" (UID: \"e1d57a75-2039-44db-abae-0085e773803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"
Nov 02 17:53:24.268156 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.267864    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d57a75-2039-44db-abae-0085e773803f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-769b70-f4ccj\" (UID: \"e1d57a75-2039-44db-abae-0085e773803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"
Nov 02 17:53:24.270053 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.269795    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d57a75-2039-44db-abae-0085e773803f-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-769b70-f4ccj\" (UID: \"e1d57a75-2039-44db-abae-0085e773803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"
Nov 02 17:53:24.274245 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.274189    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d57a75-2039-44db-abae-0085e773803f-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-769b70-f4ccj\" (UID: \"e1d57a75-2039-44db-abae-0085e773803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"
Nov 02 17:53:24.331037 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.330998    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-79zh7\" (UniqueName: \"kubernetes.io/projected/e1d57a75-2039-44db-abae-0085e773803f-kube-api-access-79zh7\") pod \"maas-api-revoke-keys-e2e-api-b-769b70-f4ccj\" (UID: \"e1d57a75-2039-44db-abae-0085e773803f\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"
Nov 02 17:53:24.385847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.385091    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"
Nov 02 17:53:24.646637 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.644626    4456 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 02 17:53:24.646637 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.644685    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql" podUID="32d7a3ef-ec4e-4639-b849-baf8078f5fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 02 17:53:24.648905 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.648882    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"]
Nov 02 17:53:24.649010 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.648908    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:24.649010 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.648974    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:24.649010 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.649007    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:53:24.844040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.843997    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"]
Nov 02 17:53:24.940710 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.940677    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp" event={"ID":"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e","Type":"ContainerStarted","Data":"1f3338eaa0ca5d0255f41a33c509df2ba8023c26afcd02725fd1c7f7c3299602"}
Nov 02 17:53:24.943200 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.943172    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" event={"ID":"0a41f8f2-1497-476f-b391-8b24c9c7f7f4","Type":"ContainerStarted","Data":"8a129e0cab5c942205679f6b457e66733b2f8b0cc96304573dab32c95c14d867"}
Nov 02 17:53:24.943325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.943222    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" event={"ID":"0a41f8f2-1497-476f-b391-8b24c9c7f7f4","Type":"ContainerStarted","Data":"c6ff67e2a366daf78949aa694518cce2d007bd0eca1783c54f56c5ce551d474d"}
Nov 02 17:53:24.946047 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:24.946028    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj" event={"ID":"e1d57a75-2039-44db-abae-0085e773803f","Type":"ContainerStarted","Data":"3ad681f941f7e5459af063afc8a03c8395e409a8c14d915a0a5cf21921143b85"}
Nov 02 17:53:25.072272 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:25.072163    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" podStartSLOduration=2.072149003 podStartE2EDuration="2.072149003s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:25.071016409 +0000 UTC m=+1421.562079595" watchObservedRunningTime="2026-11-02 17:53:25.072149003 +0000 UTC m=+1421.563212164"
Nov 02 17:53:25.218841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:25.209503    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:25.218841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:25.209630    4456 patch_prober.go:29] interesting pod/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 02 17:53:25.218841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:25.209676    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 02 17:53:25.301434 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:25.301336    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:25.476234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:25.476152    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:53:25.476641 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:25.476618    4456 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 02 17:53:25.476711 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:25.476668    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp" podUID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 02 17:53:25.644365 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:25.644272    4456 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 02 17:53:25.644662 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:25.644632    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql" podUID="32d7a3ef-ec4e-4639-b849-baf8078f5fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 02 17:53:25.961002 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:25.960944    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj" event={"ID":"e1d57a75-2039-44db-abae-0085e773803f","Type":"ContainerStarted","Data":"af497034a0769c385617fd2ca325650ebb53a06c30ab8f3d7f487ba00bc37c69"}
Nov 02 17:53:26.035862 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:26.028774    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj" podStartSLOduration=3.028753121 podStartE2EDuration="3.028753121s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:53:23 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:53:26.007112223 +0000 UTC m=+1422.498175409" watchObservedRunningTime="2026-11-02 17:53:26.028753121 +0000 UTC m=+1422.519816283"
Nov 02 17:53:26.230533 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:26.226473    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:26.476057 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:26.476022    4456 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 02 17:53:26.476206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:26.476074    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp" podUID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 02 17:53:26.650214 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:26.648533    4456 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 02 17:53:26.650214 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:26.648612    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql" podUID="32d7a3ef-ec4e-4639-b849-baf8078f5fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 02 17:53:26.971753 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:26.971715    4456 generic.go:356] "Generic (PLEG): container finished" podID="e1d57a75-2039-44db-abae-0085e773803f" containerID="af497034a0769c385617fd2ca325650ebb53a06c30ab8f3d7f487ba00bc37c69" exitCode=0
Nov 02 17:53:26.972262 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:26.971803    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj" event={"ID":"e1d57a75-2039-44db-abae-0085e773803f","Type":"ContainerDied","Data":"af497034a0769c385617fd2ca325650ebb53a06c30ab8f3d7f487ba00bc37c69"}
Nov 02 17:53:26.972262 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:26.972226    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:26.973528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:26.973507    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:53:27.479849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:27.479165    4456 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 02 17:53:27.479849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:27.479228    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp" podUID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 02 17:53:27.563239 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:27.563094    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:27.656837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:27.656158    4456 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused" start-of-body=
Nov 02 17:53:27.656837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:27.656211    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql" podUID="32d7a3ef-ec4e-4639-b849-baf8078f5fd0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.163:15021/healthz/ready\": dial tcp 10.134.0.163:15021: connect: connection refused"
Nov 02 17:53:27.767146 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:27.767055    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:27.943954 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:27.943110    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:28.439193 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.439170    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"
Nov 02 17:53:28.476473 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.476431    4456 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 02 17:53:28.476640 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.476499    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp" podUID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 02 17:53:28.564876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.560877    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e1d57a75-2039-44db-abae-0085e773803f-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d57a75-2039-44db-abae-0085e773803f-maas-api-xks-ca\") pod \"e1d57a75-2039-44db-abae-0085e773803f\" (UID: \"e1d57a75-2039-44db-abae-0085e773803f\") "
Nov 02 17:53:28.564876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.561666    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e1d57a75-2039-44db-abae-0085e773803f-kube-api-access-79zh7\" (UniqueName: \"kubernetes.io/projected/e1d57a75-2039-44db-abae-0085e773803f-kube-api-access-79zh7\") pod \"e1d57a75-2039-44db-abae-0085e773803f\" (UID: \"e1d57a75-2039-44db-abae-0085e773803f\") "
Nov 02 17:53:28.564876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.561736    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e1d57a75-2039-44db-abae-0085e773803f-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d57a75-2039-44db-abae-0085e773803f-maas-api-openshift-ca\") pod \"e1d57a75-2039-44db-abae-0085e773803f\" (UID: \"e1d57a75-2039-44db-abae-0085e773803f\") "
Nov 02 17:53:28.564876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.562244    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e1d57a75-2039-44db-abae-0085e773803f-maas-api-openshift-ca" pod "e1d57a75-2039-44db-abae-0085e773803f" (UID: "e1d57a75-2039-44db-abae-0085e773803f"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:28.564876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.563090    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e1d57a75-2039-44db-abae-0085e773803f-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:28.573940 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.565912    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e1d57a75-2039-44db-abae-0085e773803f-maas-api-xks-ca" pod "e1d57a75-2039-44db-abae-0085e773803f" (UID: "e1d57a75-2039-44db-abae-0085e773803f"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:53:28.574195 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.574156    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e1d57a75-2039-44db-abae-0085e773803f-kube-api-access-79zh7" pod "e1d57a75-2039-44db-abae-0085e773803f" (UID: "e1d57a75-2039-44db-abae-0085e773803f"). InnerVolumeSpecName "kube-api-access-79zh7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:28.667911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.667879    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-79zh7\" (UniqueName: \"kubernetes.io/projected/e1d57a75-2039-44db-abae-0085e773803f-kube-api-access-79zh7\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:28.668178 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.668153    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e1d57a75-2039-44db-abae-0085e773803f-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:28.746572 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.746401    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:28.747220 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.747194    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:28.749397 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.749346    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:53:28.773230 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.773209    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:53:28.870572 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.870540    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-workload-socket\") pod \"98c10ff3-d496-4336-a2cd-921042c719d3\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") "
Nov 02 17:53:28.870725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.870630    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-workload-certs\") pod \"98c10ff3-d496-4336-a2cd-921042c719d3\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") "
Nov 02 17:53:28.870725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.870674    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-istio-envoy\") pod \"98c10ff3-d496-4336-a2cd-921042c719d3\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") "
Nov 02 17:53:28.870725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.870706    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c10ff3-d496-4336-a2cd-921042c719d3-istio-token\" (UniqueName: \"kubernetes.io/projected/98c10ff3-d496-4336-a2cd-921042c719d3-istio-token\") pod \"98c10ff3-d496-4336-a2cd-921042c719d3\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") "
Nov 02 17:53:28.870909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.870766    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/98c10ff3-d496-4336-a2cd-921042c719d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c10ff3-d496-4336-a2cd-921042c719d3-istiod-ca-cert\") pod \"98c10ff3-d496-4336-a2cd-921042c719d3\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") "
Nov 02 17:53:28.870909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.870856    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-workload-socket" pod "98c10ff3-d496-4336-a2cd-921042c719d3" (UID: "98c10ff3-d496-4336-a2cd-921042c719d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:28.871023 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.870908    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/98c10ff3-d496-4336-a2cd-921042c719d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c10ff3-d496-4336-a2cd-921042c719d3-istio-podinfo\") pod \"98c10ff3-d496-4336-a2cd-921042c719d3\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") "
Nov 02 17:53:28.871023 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.870958    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-workload-certs" pod "98c10ff3-d496-4336-a2cd-921042c719d3" (UID: "98c10ff3-d496-4336-a2cd-921042c719d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:28.871023 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.871005    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/98c10ff3-d496-4336-a2cd-921042c719d3-kube-api-access-rctlf\" (UniqueName: \"kubernetes.io/projected/98c10ff3-d496-4336-a2cd-921042c719d3-kube-api-access-rctlf\") pod \"98c10ff3-d496-4336-a2cd-921042c719d3\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") "
Nov 02 17:53:28.871202 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.871110    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-credential-socket\") pod \"98c10ff3-d496-4336-a2cd-921042c719d3\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") "
Nov 02 17:53:28.871202 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.871170    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/98c10ff3-d496-4336-a2cd-921042c719d3-istiod-ca-cert" pod "98c10ff3-d496-4336-a2cd-921042c719d3" (UID: "98c10ff3-d496-4336-a2cd-921042c719d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:28.871202 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.871188    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-istio-data\") pod \"98c10ff3-d496-4336-a2cd-921042c719d3\" (UID: \"98c10ff3-d496-4336-a2cd-921042c719d3\") "
Nov 02 17:53:28.871390 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.871362    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-credential-socket" pod "98c10ff3-d496-4336-a2cd-921042c719d3" (UID: "98c10ff3-d496-4336-a2cd-921042c719d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:28.871737 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.871649    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-istio-data" pod "98c10ff3-d496-4336-a2cd-921042c719d3" (UID: "98c10ff3-d496-4336-a2cd-921042c719d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:28.872135 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.872111    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:28.872217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.872141    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:28.872217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.872158    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/98c10ff3-d496-4336-a2cd-921042c719d3-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:28.872217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.872170    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:28.872217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.872185    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:28.873128 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.873107    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c10ff3-d496-4336-a2cd-921042c719d3-istio-token" pod "98c10ff3-d496-4336-a2cd-921042c719d3" (UID: "98c10ff3-d496-4336-a2cd-921042c719d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:28.873213 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.873194    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-istio-envoy" pod "98c10ff3-d496-4336-a2cd-921042c719d3" (UID: "98c10ff3-d496-4336-a2cd-921042c719d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:28.873275 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.873251    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/98c10ff3-d496-4336-a2cd-921042c719d3-istio-podinfo" pod "98c10ff3-d496-4336-a2cd-921042c719d3" (UID: "98c10ff3-d496-4336-a2cd-921042c719d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:53:28.873424 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.873407    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/98c10ff3-d496-4336-a2cd-921042c719d3-kube-api-access-rctlf" pod "98c10ff3-d496-4336-a2cd-921042c719d3" (UID: "98c10ff3-d496-4336-a2cd-921042c719d3"). InnerVolumeSpecName "kube-api-access-rctlf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:28.975265 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.974422    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rctlf\" (UniqueName: \"kubernetes.io/projected/98c10ff3-d496-4336-a2cd-921042c719d3-kube-api-access-rctlf\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:28.975265 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.974456    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/98c10ff3-d496-4336-a2cd-921042c719d3-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:28.975265 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.974472    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/98c10ff3-d496-4336-a2cd-921042c719d3-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:28.975265 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:28.974485    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/98c10ff3-d496-4336-a2cd-921042c719d3-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:29.029903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.029787    4456 generic.go:356] "Generic (PLEG): container finished" podID="98c10ff3-d496-4336-a2cd-921042c719d3" containerID="77600a881963b6c995f4d2afec219dad9a8788b39af4a147947ed27a989fad54" exitCode=0
Nov 02 17:53:29.029903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.029863    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"
Nov 02 17:53:29.030132 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.029914    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7" event={"ID":"98c10ff3-d496-4336-a2cd-921042c719d3","Type":"ContainerDied","Data":"77600a881963b6c995f4d2afec219dad9a8788b39af4a147947ed27a989fad54"}
Nov 02 17:53:29.030132 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.029952    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7" event={"ID":"98c10ff3-d496-4336-a2cd-921042c719d3","Type":"ContainerDied","Data":"89e536a98227e233f0423debcecaf77140ae2c4f77dd6711832ca28d5bfba7b5"}
Nov 02 17:53:29.030132 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.029972    4456 scope.go:122] "RemoveContainer" containerID="77600a881963b6c995f4d2afec219dad9a8788b39af4a147947ed27a989fad54"
Nov 02 17:53:29.032184 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.032154    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj" event={"ID":"e1d57a75-2039-44db-abae-0085e773803f","Type":"ContainerDied","Data":"3ad681f941f7e5459af063afc8a03c8395e409a8c14d915a0a5cf21921143b85"}
Nov 02 17:53:29.032297 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.032207    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ad681f941f7e5459af063afc8a03c8395e409a8c14d915a0a5cf21921143b85"
Nov 02 17:53:29.032705 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.032686    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"
Nov 02 17:53:29.061756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.061729    4456 scope.go:122] "RemoveContainer" containerID="4c2b669d919f440b588554d96ba166ca71127cc37440ec0db956fe5315cc5fdc"
Nov 02 17:53:29.073612 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.073590    4456 scope.go:122] "RemoveContainer" containerID="77600a881963b6c995f4d2afec219dad9a8788b39af4a147947ed27a989fad54"
Nov 02 17:53:29.073961 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:53:29.073939    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"77600a881963b6c995f4d2afec219dad9a8788b39af4a147947ed27a989fad54\": container with ID starting with 77600a881963b6c995f4d2afec219dad9a8788b39af4a147947ed27a989fad54 not found: ID does not exist" containerID="77600a881963b6c995f4d2afec219dad9a8788b39af4a147947ed27a989fad54"
Nov 02 17:53:29.074062 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.073970    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"77600a881963b6c995f4d2afec219dad9a8788b39af4a147947ed27a989fad54"} err="failed to get container status \"77600a881963b6c995f4d2afec219dad9a8788b39af4a147947ed27a989fad54\": rpc error: code = NotFound desc = could not find container \"77600a881963b6c995f4d2afec219dad9a8788b39af4a147947ed27a989fad54\": container with ID starting with 77600a881963b6c995f4d2afec219dad9a8788b39af4a147947ed27a989fad54 not found: ID does not exist"
Nov 02 17:53:29.074062 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.073988    4456 scope.go:122] "RemoveContainer" containerID="4c2b669d919f440b588554d96ba166ca71127cc37440ec0db956fe5315cc5fdc"
Nov 02 17:53:29.074269 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:53:29.074250    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4c2b669d919f440b588554d96ba166ca71127cc37440ec0db956fe5315cc5fdc\": container with ID starting with 4c2b669d919f440b588554d96ba166ca71127cc37440ec0db956fe5315cc5fdc not found: ID does not exist" containerID="4c2b669d919f440b588554d96ba166ca71127cc37440ec0db956fe5315cc5fdc"
Nov 02 17:53:29.074324 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.074278    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4c2b669d919f440b588554d96ba166ca71127cc37440ec0db956fe5315cc5fdc"} err="failed to get container status \"4c2b669d919f440b588554d96ba166ca71127cc37440ec0db956fe5315cc5fdc\": rpc error: code = NotFound desc = could not find container \"4c2b669d919f440b588554d96ba166ca71127cc37440ec0db956fe5315cc5fdc\": container with ID starting with 4c2b669d919f440b588554d96ba166ca71127cc37440ec0db956fe5315cc5fdc not found: ID does not exist"
Nov 02 17:53:29.144660 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.144625    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"]
Nov 02 17:53:29.148787 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.148763    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7"]
Nov 02 17:53:29.476389 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.476351    4456 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 02 17:53:29.477010 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.476420    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp" podUID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 02 17:53:29.518858 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.518730    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:29.631715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.631631    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="98c10ff3-d496-4336-a2cd-921042c719d3" path="/var/lib/kubelet/pods/98c10ff3-d496-4336-a2cd-921042c719d3/volumes"
Nov 02 17:53:29.683840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.679916    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:29.759839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.754916    4456 patch_prober.go:29] interesting pod/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:53:29.759839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:29.754986    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-f99b8b-openshift-default-85c4d77577-f65b7" podUID="98c10ff3-d496-4336-a2cd-921042c719d3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.136:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:53:30.476485 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:30.476449    4456 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused" start-of-body=
Nov 02 17:53:30.476966 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:30.476503    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp" podUID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.153:15021/healthz/ready\": dial tcp 10.134.0.153:15021: connect: connection refused"
Nov 02 17:53:31.357018 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:31.356969    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:31.489491 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:31.488035    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:53:31.490007 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:31.489914    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:53:31.491104 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:31.491082    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:53:33.133952 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:33.126043    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:34.455119 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:34.455085    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"]
Nov 02 17:53:34.489072 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:34.489037    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"]
Nov 02 17:53:34.634830 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:34.634783    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:34.774501 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:34.774421    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:35.265327 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:35.265291    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:36.493511 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:36.493399    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:38.355167 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:38.355125    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:39.455584 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:39.455531    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj" podUID="d6b00cb5-ecfc-4140-bb2f-2b6cd645d538" containerName="payload-processing" containerID="cri-o://b0d320f2d9eea81b8d6b8e91983823809d6484978da046f73f25f810f3f9be51" gracePeriod=25
Nov 02 17:53:39.490205 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:39.490163    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc" podUID="caf4d660-cd9a-4cd9-90bd-4ec5838649ee" containerName="payload-pre-processing" containerID="cri-o://f750a28626bdbfc0688b5b8e129b48a985473b483835c3aaf487c933e2f50a49" gracePeriod=25
Nov 02 17:53:39.836597 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:39.836562    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"
Nov 02 17:53:39.892906 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:39.892880    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"
Nov 02 17:53:39.962872 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:39.962795    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d6b00cb5-ecfc-4140-bb2f-2b6cd645d538-kube-api-access-sw6ht\" (UniqueName: \"kubernetes.io/projected/d6b00cb5-ecfc-4140-bb2f-2b6cd645d538-kube-api-access-sw6ht\") pod \"d6b00cb5-ecfc-4140-bb2f-2b6cd645d538\" (UID: \"d6b00cb5-ecfc-4140-bb2f-2b6cd645d538\") "
Nov 02 17:53:39.963053 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:39.962896    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d6b00cb5-ecfc-4140-bb2f-2b6cd645d538-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b00cb5-ecfc-4140-bb2f-2b6cd645d538-plugins-config-volume\") pod \"d6b00cb5-ecfc-4140-bb2f-2b6cd645d538\" (UID: \"d6b00cb5-ecfc-4140-bb2f-2b6cd645d538\") "
Nov 02 17:53:39.964085 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:39.964055    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d6b00cb5-ecfc-4140-bb2f-2b6cd645d538-plugins-config-volume" pod "d6b00cb5-ecfc-4140-bb2f-2b6cd645d538" (UID: "d6b00cb5-ecfc-4140-bb2f-2b6cd645d538"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:39.965440 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:39.965419    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d6b00cb5-ecfc-4140-bb2f-2b6cd645d538-kube-api-access-sw6ht" pod "d6b00cb5-ecfc-4140-bb2f-2b6cd645d538" (UID: "d6b00cb5-ecfc-4140-bb2f-2b6cd645d538"). InnerVolumeSpecName "kube-api-access-sw6ht". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:40.065119 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.065015    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/caf4d660-cd9a-4cd9-90bd-4ec5838649ee-kube-api-access-sqtnr\" (UniqueName: \"kubernetes.io/projected/caf4d660-cd9a-4cd9-90bd-4ec5838649ee-kube-api-access-sqtnr\") pod \"caf4d660-cd9a-4cd9-90bd-4ec5838649ee\" (UID: \"caf4d660-cd9a-4cd9-90bd-4ec5838649ee\") "
Nov 02 17:53:40.065299 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.065217    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/caf4d660-cd9a-4cd9-90bd-4ec5838649ee-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf4d660-cd9a-4cd9-90bd-4ec5838649ee-plugins-config-volume\") pod \"caf4d660-cd9a-4cd9-90bd-4ec5838649ee\" (UID: \"caf4d660-cd9a-4cd9-90bd-4ec5838649ee\") "
Nov 02 17:53:40.066152 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.066118    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sw6ht\" (UniqueName: \"kubernetes.io/projected/d6b00cb5-ecfc-4140-bb2f-2b6cd645d538-kube-api-access-sw6ht\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:40.066152 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.066148    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d6b00cb5-ecfc-4140-bb2f-2b6cd645d538-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:40.069163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.069119    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/caf4d660-cd9a-4cd9-90bd-4ec5838649ee-plugins-config-volume" pod "caf4d660-cd9a-4cd9-90bd-4ec5838649ee" (UID: "caf4d660-cd9a-4cd9-90bd-4ec5838649ee"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:40.105839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.079339    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/caf4d660-cd9a-4cd9-90bd-4ec5838649ee-kube-api-access-sqtnr" pod "caf4d660-cd9a-4cd9-90bd-4ec5838649ee" (UID: "caf4d660-cd9a-4cd9-90bd-4ec5838649ee"). InnerVolumeSpecName "kube-api-access-sqtnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:40.180670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.180633    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sqtnr\" (UniqueName: \"kubernetes.io/projected/caf4d660-cd9a-4cd9-90bd-4ec5838649ee-kube-api-access-sqtnr\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:40.181410 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.181385    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/caf4d660-cd9a-4cd9-90bd-4ec5838649ee-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:40.234842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.232784    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:40.313344 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.313307    4456 generic.go:356] "Generic (PLEG): container finished" podID="d6b00cb5-ecfc-4140-bb2f-2b6cd645d538" containerID="b0d320f2d9eea81b8d6b8e91983823809d6484978da046f73f25f810f3f9be51" exitCode=0
Nov 02 17:53:40.313538 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.313408    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj" event={"ID":"d6b00cb5-ecfc-4140-bb2f-2b6cd645d538","Type":"ContainerDied","Data":"b0d320f2d9eea81b8d6b8e91983823809d6484978da046f73f25f810f3f9be51"}
Nov 02 17:53:40.313538 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.313460    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj" event={"ID":"d6b00cb5-ecfc-4140-bb2f-2b6cd645d538","Type":"ContainerDied","Data":"72f833f70c48a66e3b942031121c20db81be5708019e9a69d1109b73766be403"}
Nov 02 17:53:40.313538 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.313489    4456 scope.go:122] "RemoveContainer" containerID="b0d320f2d9eea81b8d6b8e91983823809d6484978da046f73f25f810f3f9be51"
Nov 02 17:53:40.314409 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.314387    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"
Nov 02 17:53:40.318429 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.317961    4456 generic.go:356] "Generic (PLEG): container finished" podID="caf4d660-cd9a-4cd9-90bd-4ec5838649ee" containerID="f750a28626bdbfc0688b5b8e129b48a985473b483835c3aaf487c933e2f50a49" exitCode=0
Nov 02 17:53:40.328717 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.324385    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"
Nov 02 17:53:40.328717 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.324710    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc" event={"ID":"caf4d660-cd9a-4cd9-90bd-4ec5838649ee","Type":"ContainerDied","Data":"f750a28626bdbfc0688b5b8e129b48a985473b483835c3aaf487c933e2f50a49"}
Nov 02 17:53:40.328717 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.324893    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc" event={"ID":"caf4d660-cd9a-4cd9-90bd-4ec5838649ee","Type":"ContainerDied","Data":"259cdef8ec0bb0ba31b0808afe9fae14dc4554c07dbcb6269ce360a21ebd1e84"}
Nov 02 17:53:40.350970 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.350944    4456 scope.go:122] "RemoveContainer" containerID="b0d320f2d9eea81b8d6b8e91983823809d6484978da046f73f25f810f3f9be51"
Nov 02 17:53:40.351431 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:53:40.351404    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b0d320f2d9eea81b8d6b8e91983823809d6484978da046f73f25f810f3f9be51\": container with ID starting with b0d320f2d9eea81b8d6b8e91983823809d6484978da046f73f25f810f3f9be51 not found: ID does not exist" containerID="b0d320f2d9eea81b8d6b8e91983823809d6484978da046f73f25f810f3f9be51"
Nov 02 17:53:40.351542 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.351439    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b0d320f2d9eea81b8d6b8e91983823809d6484978da046f73f25f810f3f9be51"} err="failed to get container status \"b0d320f2d9eea81b8d6b8e91983823809d6484978da046f73f25f810f3f9be51\": rpc error: code = NotFound desc = could not find container \"b0d320f2d9eea81b8d6b8e91983823809d6484978da046f73f25f810f3f9be51\": container with ID starting with b0d320f2d9eea81b8d6b8e91983823809d6484978da046f73f25f810f3f9be51 not found: ID does not exist"
Nov 02 17:53:40.351542 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.351458    4456 scope.go:122] "RemoveContainer" containerID="f750a28626bdbfc0688b5b8e129b48a985473b483835c3aaf487c933e2f50a49"
Nov 02 17:53:40.372445 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.372293    4456 scope.go:122] "RemoveContainer" containerID="f750a28626bdbfc0688b5b8e129b48a985473b483835c3aaf487c933e2f50a49"
Nov 02 17:53:40.373456 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:53:40.373427    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f750a28626bdbfc0688b5b8e129b48a985473b483835c3aaf487c933e2f50a49\": container with ID starting with f750a28626bdbfc0688b5b8e129b48a985473b483835c3aaf487c933e2f50a49 not found: ID does not exist" containerID="f750a28626bdbfc0688b5b8e129b48a985473b483835c3aaf487c933e2f50a49"
Nov 02 17:53:40.374320 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.374281    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f750a28626bdbfc0688b5b8e129b48a985473b483835c3aaf487c933e2f50a49"} err="failed to get container status \"f750a28626bdbfc0688b5b8e129b48a985473b483835c3aaf487c933e2f50a49\": rpc error: code = NotFound desc = could not find container \"f750a28626bdbfc0688b5b8e129b48a985473b483835c3aaf487c933e2f50a49\": container with ID starting with f750a28626bdbfc0688b5b8e129b48a985473b483835c3aaf487c933e2f50a49 not found: ID does not exist"
Nov 02 17:53:40.399864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.399632    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"]
Nov 02 17:53:40.439942 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.439899    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-api-b-769b70-67967c7fdc-mx8qc"]
Nov 02 17:53:40.447368 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.447338    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"]
Nov 02 17:53:40.451900 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:40.451866    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-api-b-769b70-bf6977767-nwxwj"]
Nov 02 17:53:41.624917 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:41.624874    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="caf4d660-cd9a-4cd9-90bd-4ec5838649ee" path="/var/lib/kubelet/pods/caf4d660-cd9a-4cd9-90bd-4ec5838649ee/volumes"
Nov 02 17:53:41.625662 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:41.625636    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d6b00cb5-ecfc-4140-bb2f-2b6cd645d538" path="/var/lib/kubelet/pods/d6b00cb5-ecfc-4140-bb2f-2b6cd645d538/volumes"
Nov 02 17:53:41.708023 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:41.707978    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:42.322838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:42.320415    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:42.467109 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:42.467068    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:42.481912 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:42.477158    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"]
Nov 02 17:53:42.521911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:42.515649    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-786c42-6tcn8"]
Nov 02 17:53:42.998596 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:42.998556    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:43.180376 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:43.180343    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:43.833840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:43.832858    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:43.914839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:43.890028    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d045ce96-3b4a-4f2b-b928-a30bd681dcb3" path="/var/lib/kubelet/pods/d045ce96-3b4a-4f2b-b928-a30bd681dcb3/volumes"
Nov 02 17:53:44.372656 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:44.372615    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:44.632532 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:44.630207    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:46.144752 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:46.144723    4456 scope.go:122] "RemoveContainer" containerID="8840ba33147f6d7c04b50c022624be7e756f0b168b4f6c6c496dc8e5f954fe01"
Nov 02 17:53:46.157003 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:46.156981    4456 scope.go:122] "RemoveContainer" containerID="343e10261cea507061afeab13d56732972ca7a49e6face19ff3c2566f889aa49"
Nov 02 17:53:46.168315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:46.168294    4456 scope.go:122] "RemoveContainer" containerID="ca36d520ce7140765fc905f1273ef62ab673d6854b596a5f11b5a47473a12e51"
Nov 02 17:53:46.174432 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:53:46.174400    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:53:46.430132 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:46.429983    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:46.577082 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:46.577042    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:46.716843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:46.715801    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:47.612837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:47.602146    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"]
Nov 02 17:53:47.641909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:47.641865    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2" podUID="20a26856-e0a4-410b-978b-4c65e143a80b" containerName="istio-proxy" containerID="cri-o://36d3f466742bc004aa1bcccf3f09c43559dd6325d3ef0d486dba3840c5ffaf38" gracePeriod=30
Nov 02 17:53:47.673337 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:47.673069    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:47.826142 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:47.826110    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:48.136473 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:48.136433    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"]
Nov 02 17:53:48.136686 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:48.136661    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc" podUID="363eabf7-2a32-4f2c-9333-d03592a4b41d" containerName="istio-proxy" containerID="cri-o://0bcd336e8308ef139181ab90e2f3233572e19d0a9c8c55bb55e13405958b21ef" gracePeriod=30
Nov 02 17:53:48.558714 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:48.558675    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:49.437311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:49.437279    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:49.581998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:49.581961    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:50.218362 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.218331    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"]
Nov 02 17:53:50.219148 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.219130    4456 cpu_manager.go:414] "containerMap: removing container" podUID="98c10ff3-d496-4336-a2cd-921042c719d3" containerName="istio-proxy"
Nov 02 17:53:50.219148 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.219149    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c10ff3-d496-4336-a2cd-921042c719d3" containerName="istio-proxy"
Nov 02 17:53:50.219248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.219164    4456 cpu_manager.go:414] "containerMap: removing container" podUID="98c10ff3-d496-4336-a2cd-921042c719d3" containerName="istio-proxy"
Nov 02 17:53:50.219248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.219169    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="98c10ff3-d496-4336-a2cd-921042c719d3" containerName="istio-proxy"
Nov 02 17:53:50.219248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.219193    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e1d57a75-2039-44db-abae-0085e773803f" containerName="revoke-keys"
Nov 02 17:53:50.219248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.219199    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e1d57a75-2039-44db-abae-0085e773803f" containerName="revoke-keys"
Nov 02 17:53:50.219248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.219209    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d6b00cb5-ecfc-4140-bb2f-2b6cd645d538" containerName="payload-processing"
Nov 02 17:53:50.219248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.219214    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d6b00cb5-ecfc-4140-bb2f-2b6cd645d538" containerName="payload-processing"
Nov 02 17:53:50.219248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.219239    4456 cpu_manager.go:414] "containerMap: removing container" podUID="caf4d660-cd9a-4cd9-90bd-4ec5838649ee" containerName="payload-pre-processing"
Nov 02 17:53:50.219248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.219244    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="caf4d660-cd9a-4cd9-90bd-4ec5838649ee" containerName="payload-pre-processing"
Nov 02 17:53:50.219490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.219349    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="caf4d660-cd9a-4cd9-90bd-4ec5838649ee" containerName="payload-pre-processing"
Nov 02 17:53:50.219490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.219361    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c10ff3-d496-4336-a2cd-921042c719d3" containerName="istio-proxy"
Nov 02 17:53:50.219490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.219370    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="98c10ff3-d496-4336-a2cd-921042c719d3" containerName="istio-proxy"
Nov 02 17:53:50.219490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.219379    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e1d57a75-2039-44db-abae-0085e773803f" containerName="revoke-keys"
Nov 02 17:53:50.219490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.219385    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d6b00cb5-ecfc-4140-bb2f-2b6cd645d538" containerName="payload-processing"
Nov 02 17:53:50.225259 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.225240    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"
Nov 02 17:53:50.240612 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.240587    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"]
Nov 02 17:53:50.322129 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.322099    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:53:50.322313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.322212    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:53:50.347468 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.347433    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tv85l\" (UniqueName: \"kubernetes.io/projected/eb819cb7-0963-4c99-91c4-24ac65143c75-kube-api-access-tv85l\") pod \"maas-api-revoke-sub-6744769d-ts85l\" (UID: \"eb819cb7-0963-4c99-91c4-24ac65143c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"
Nov 02 17:53:50.347584 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.347564    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb819cb7-0963-4c99-91c4-24ac65143c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6744769d-ts85l\" (UID: \"eb819cb7-0963-4c99-91c4-24ac65143c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"
Nov 02 17:53:50.347723 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.347710    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb819cb7-0963-4c99-91c4-24ac65143c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6744769d-ts85l\" (UID: \"eb819cb7-0963-4c99-91c4-24ac65143c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"
Nov 02 17:53:50.449581 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.449545    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb819cb7-0963-4c99-91c4-24ac65143c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6744769d-ts85l\" (UID: \"eb819cb7-0963-4c99-91c4-24ac65143c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"
Nov 02 17:53:50.449993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.449675    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb819cb7-0963-4c99-91c4-24ac65143c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6744769d-ts85l\" (UID: \"eb819cb7-0963-4c99-91c4-24ac65143c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"
Nov 02 17:53:50.449993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.449870    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tv85l\" (UniqueName: \"kubernetes.io/projected/eb819cb7-0963-4c99-91c4-24ac65143c75-kube-api-access-tv85l\") pod \"maas-api-revoke-sub-6744769d-ts85l\" (UID: \"eb819cb7-0963-4c99-91c4-24ac65143c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"
Nov 02 17:53:50.450236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.450209    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb819cb7-0963-4c99-91c4-24ac65143c75-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6744769d-ts85l\" (UID: \"eb819cb7-0963-4c99-91c4-24ac65143c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"
Nov 02 17:53:50.452266 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.452236    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb819cb7-0963-4c99-91c4-24ac65143c75-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6744769d-ts85l\" (UID: \"eb819cb7-0963-4c99-91c4-24ac65143c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"
Nov 02 17:53:50.474086 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.474020    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tv85l\" (UniqueName: \"kubernetes.io/projected/eb819cb7-0963-4c99-91c4-24ac65143c75-kube-api-access-tv85l\") pod \"maas-api-revoke-sub-6744769d-ts85l\" (UID: \"eb819cb7-0963-4c99-91c4-24ac65143c75\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"
Nov 02 17:53:50.546213 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.546182    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"
Nov 02 17:53:50.671419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.670052    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:50.839713 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.839627    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:50.871043 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.871011    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"]
Nov 02 17:53:50.977975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:50.977942    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:51.272894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:51.272856    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"]
Nov 02 17:53:51.273137 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:51.273107    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd" podUID="33530571-d61c-488e-be5b-54e298cfd4e4" containerName="istio-proxy" containerID="cri-o://85b38081a920ac04873b6d419f556f5ffffef86b5115b351b3b2b35a26058e83" gracePeriod=30
Nov 02 17:53:51.312126 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:51.312092    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:51.488968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:51.477347    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l" event={"ID":"eb819cb7-0963-4c99-91c4-24ac65143c75","Type":"ContainerStarted","Data":"07ed931c42a138643003e09c77c29459fe28ec2af21909f685596a20e1fc4ebf"}
Nov 02 17:53:52.487642 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:52.487604    4456 generic.go:356] "Generic (PLEG): container finished" podID="eb819cb7-0963-4c99-91c4-24ac65143c75" containerID="d376b250cd604ac4f7fa5e98f47fea37e0b793fc6d46a3b9eb9b4ab25fce9a79" exitCode=0
Nov 02 17:53:52.487901 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:52.487687    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l" event={"ID":"eb819cb7-0963-4c99-91c4-24ac65143c75","Type":"ContainerDied","Data":"d376b250cd604ac4f7fa5e98f47fea37e0b793fc6d46a3b9eb9b4ab25fce9a79"}
Nov 02 17:53:52.693587 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:52.693554    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:52.978280 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:52.978256    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:53:53.073496 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.073398    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-istio-data\") pod \"20a26856-e0a4-410b-978b-4c65e143a80b\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") "
Nov 02 17:53:53.073496 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.073486    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-workload-certs\") pod \"20a26856-e0a4-410b-978b-4c65e143a80b\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") "
Nov 02 17:53:53.073735 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.073575    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-istio-envoy\") pod \"20a26856-e0a4-410b-978b-4c65e143a80b\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") "
Nov 02 17:53:53.073735 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.073684    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a26856-e0a4-410b-978b-4c65e143a80b-kube-api-access-lvkmg\" (UniqueName: \"kubernetes.io/projected/20a26856-e0a4-410b-978b-4c65e143a80b-kube-api-access-lvkmg\") pod \"20a26856-e0a4-410b-978b-4c65e143a80b\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") "
Nov 02 17:53:53.073735 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.073694    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-istio-data" pod "20a26856-e0a4-410b-978b-4c65e143a80b" (UID: "20a26856-e0a4-410b-978b-4c65e143a80b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:53.073735 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.073721    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-workload-certs" pod "20a26856-e0a4-410b-978b-4c65e143a80b" (UID: "20a26856-e0a4-410b-978b-4c65e143a80b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:53.073993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.073760    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-credential-socket\") pod \"20a26856-e0a4-410b-978b-4c65e143a80b\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") "
Nov 02 17:53:53.073993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.073803    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/20a26856-e0a4-410b-978b-4c65e143a80b-istio-token\" (UniqueName: \"kubernetes.io/projected/20a26856-e0a4-410b-978b-4c65e143a80b-istio-token\") pod \"20a26856-e0a4-410b-978b-4c65e143a80b\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") "
Nov 02 17:53:53.073993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.073848    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/20a26856-e0a4-410b-978b-4c65e143a80b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a26856-e0a4-410b-978b-4c65e143a80b-istio-podinfo\") pod \"20a26856-e0a4-410b-978b-4c65e143a80b\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") "
Nov 02 17:53:53.073993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.073909    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-workload-socket\") pod \"20a26856-e0a4-410b-978b-4c65e143a80b\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") "
Nov 02 17:53:53.073993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.073938    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/20a26856-e0a4-410b-978b-4c65e143a80b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a26856-e0a4-410b-978b-4c65e143a80b-istiod-ca-cert\") pod \"20a26856-e0a4-410b-978b-4c65e143a80b\" (UID: \"20a26856-e0a4-410b-978b-4c65e143a80b\") "
Nov 02 17:53:53.074256 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.073998    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-credential-socket" pod "20a26856-e0a4-410b-978b-4c65e143a80b" (UID: "20a26856-e0a4-410b-978b-4c65e143a80b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:53.074506 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.074485    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.074628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.074508    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.074628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.074521    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.074863 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.074846    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/20a26856-e0a4-410b-978b-4c65e143a80b-istiod-ca-cert" pod "20a26856-e0a4-410b-978b-4c65e143a80b" (UID: "20a26856-e0a4-410b-978b-4c65e143a80b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:53.075064 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.075045    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-workload-socket" pod "20a26856-e0a4-410b-978b-4c65e143a80b" (UID: "20a26856-e0a4-410b-978b-4c65e143a80b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:53.076691 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.076669    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/20a26856-e0a4-410b-978b-4c65e143a80b-istio-podinfo" pod "20a26856-e0a4-410b-978b-4c65e143a80b" (UID: "20a26856-e0a4-410b-978b-4c65e143a80b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:53:53.076886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.076857    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a26856-e0a4-410b-978b-4c65e143a80b-istio-token" pod "20a26856-e0a4-410b-978b-4c65e143a80b" (UID: "20a26856-e0a4-410b-978b-4c65e143a80b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:53.079187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.079163    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/20a26856-e0a4-410b-978b-4c65e143a80b-kube-api-access-lvkmg" pod "20a26856-e0a4-410b-978b-4c65e143a80b" (UID: "20a26856-e0a4-410b-978b-4c65e143a80b"). InnerVolumeSpecName "kube-api-access-lvkmg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:53.087861 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.080572    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-istio-envoy" pod "20a26856-e0a4-410b-978b-4c65e143a80b" (UID: "20a26856-e0a4-410b-978b-4c65e143a80b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:53.176295 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.176271    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.176383 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.176298    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lvkmg\" (UniqueName: \"kubernetes.io/projected/20a26856-e0a4-410b-978b-4c65e143a80b-kube-api-access-lvkmg\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.176383 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.176308    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/20a26856-e0a4-410b-978b-4c65e143a80b-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.176383 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.176317    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/20a26856-e0a4-410b-978b-4c65e143a80b-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.176383 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.176325    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/20a26856-e0a4-410b-978b-4c65e143a80b-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.176383 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.176333    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/20a26856-e0a4-410b-978b-4c65e143a80b-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.423489 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.423457    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:53:53.505804 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.505764    4456 generic.go:356] "Generic (PLEG): container finished" podID="363eabf7-2a32-4f2c-9333-d03592a4b41d" containerID="0bcd336e8308ef139181ab90e2f3233572e19d0a9c8c55bb55e13405958b21ef" exitCode=0
Nov 02 17:53:53.506128 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.505856    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc" event={"ID":"363eabf7-2a32-4f2c-9333-d03592a4b41d","Type":"ContainerDied","Data":"0bcd336e8308ef139181ab90e2f3233572e19d0a9c8c55bb55e13405958b21ef"}
Nov 02 17:53:53.506128 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.505927    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc" event={"ID":"363eabf7-2a32-4f2c-9333-d03592a4b41d","Type":"ContainerDied","Data":"b0cd430826ce1adcaf0c9a7caa94e2e6db5be1fe53b8bdf66863535190c443e9"}
Nov 02 17:53:53.506128 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.505958    4456 scope.go:122] "RemoveContainer" containerID="0bcd336e8308ef139181ab90e2f3233572e19d0a9c8c55bb55e13405958b21ef"
Nov 02 17:53:53.506128 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.505867    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"
Nov 02 17:53:53.509328 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.509305    4456 generic.go:356] "Generic (PLEG): container finished" podID="20a26856-e0a4-410b-978b-4c65e143a80b" containerID="36d3f466742bc004aa1bcccf3f09c43559dd6325d3ef0d486dba3840c5ffaf38" exitCode=0
Nov 02 17:53:53.509852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.509803    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"
Nov 02 17:53:53.514748 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.513896    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2" event={"ID":"20a26856-e0a4-410b-978b-4c65e143a80b","Type":"ContainerDied","Data":"36d3f466742bc004aa1bcccf3f09c43559dd6325d3ef0d486dba3840c5ffaf38"}
Nov 02 17:53:53.514748 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.513953    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2" event={"ID":"20a26856-e0a4-410b-978b-4c65e143a80b","Type":"ContainerDied","Data":"2260b77e9d5b669a6eb507e90dbf5b42c7b883d214098d7687befbe44f307a0e"}
Nov 02 17:53:53.527252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.525093    4456 scope.go:122] "RemoveContainer" containerID="0bcd336e8308ef139181ab90e2f3233572e19d0a9c8c55bb55e13405958b21ef"
Nov 02 17:53:53.527252 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:53:53.525449    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0bcd336e8308ef139181ab90e2f3233572e19d0a9c8c55bb55e13405958b21ef\": container with ID starting with 0bcd336e8308ef139181ab90e2f3233572e19d0a9c8c55bb55e13405958b21ef not found: ID does not exist" containerID="0bcd336e8308ef139181ab90e2f3233572e19d0a9c8c55bb55e13405958b21ef"
Nov 02 17:53:53.527252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.525484    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0bcd336e8308ef139181ab90e2f3233572e19d0a9c8c55bb55e13405958b21ef"} err="failed to get container status \"0bcd336e8308ef139181ab90e2f3233572e19d0a9c8c55bb55e13405958b21ef\": rpc error: code = NotFound desc = could not find container \"0bcd336e8308ef139181ab90e2f3233572e19d0a9c8c55bb55e13405958b21ef\": container with ID starting with 0bcd336e8308ef139181ab90e2f3233572e19d0a9c8c55bb55e13405958b21ef not found: ID does not exist"
Nov 02 17:53:53.527252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.525508    4456 scope.go:122] "RemoveContainer" containerID="36d3f466742bc004aa1bcccf3f09c43559dd6325d3ef0d486dba3840c5ffaf38"
Nov 02 17:53:53.544252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.544226    4456 scope.go:122] "RemoveContainer" containerID="36d3f466742bc004aa1bcccf3f09c43559dd6325d3ef0d486dba3840c5ffaf38"
Nov 02 17:53:53.544661 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:53:53.544617    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"36d3f466742bc004aa1bcccf3f09c43559dd6325d3ef0d486dba3840c5ffaf38\": container with ID starting with 36d3f466742bc004aa1bcccf3f09c43559dd6325d3ef0d486dba3840c5ffaf38 not found: ID does not exist" containerID="36d3f466742bc004aa1bcccf3f09c43559dd6325d3ef0d486dba3840c5ffaf38"
Nov 02 17:53:53.544748 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.544662    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"36d3f466742bc004aa1bcccf3f09c43559dd6325d3ef0d486dba3840c5ffaf38"} err="failed to get container status \"36d3f466742bc004aa1bcccf3f09c43559dd6325d3ef0d486dba3840c5ffaf38\": rpc error: code = NotFound desc = could not find container \"36d3f466742bc004aa1bcccf3f09c43559dd6325d3ef0d486dba3840c5ffaf38\": container with ID starting with 36d3f466742bc004aa1bcccf3f09c43559dd6325d3ef0d486dba3840c5ffaf38 not found: ID does not exist"
Nov 02 17:53:53.591845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.590307    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"]
Nov 02 17:53:53.591845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.590708    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/363eabf7-2a32-4f2c-9333-d03592a4b41d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363eabf7-2a32-4f2c-9333-d03592a4b41d-istiod-ca-cert\") pod \"363eabf7-2a32-4f2c-9333-d03592a4b41d\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") "
Nov 02 17:53:53.591845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.590921    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-data\") pod \"363eabf7-2a32-4f2c-9333-d03592a4b41d\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") "
Nov 02 17:53:53.591845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.590953    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-workload-socket\") pod \"363eabf7-2a32-4f2c-9333-d03592a4b41d\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") "
Nov 02 17:53:53.591845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.591016    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-credential-socket\") pod \"363eabf7-2a32-4f2c-9333-d03592a4b41d\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") "
Nov 02 17:53:53.591845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.591084    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-workload-certs\") pod \"363eabf7-2a32-4f2c-9333-d03592a4b41d\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") "
Nov 02 17:53:53.591845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.591126    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-envoy\") pod \"363eabf7-2a32-4f2c-9333-d03592a4b41d\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") "
Nov 02 17:53:53.591845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.591195    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-podinfo\") pod \"363eabf7-2a32-4f2c-9333-d03592a4b41d\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") "
Nov 02 17:53:53.602911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.596403    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w6-b91583-openshift-default-b977589dc-ffkg2"]
Nov 02 17:53:53.602911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.596702    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-workload-socket" pod "363eabf7-2a32-4f2c-9333-d03592a4b41d" (UID: "363eabf7-2a32-4f2c-9333-d03592a4b41d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:53.602911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.597061    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/363eabf7-2a32-4f2c-9333-d03592a4b41d-istiod-ca-cert" pod "363eabf7-2a32-4f2c-9333-d03592a4b41d" (UID: "363eabf7-2a32-4f2c-9333-d03592a4b41d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:53.602911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.597066    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-data" pod "363eabf7-2a32-4f2c-9333-d03592a4b41d" (UID: "363eabf7-2a32-4f2c-9333-d03592a4b41d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:53.602911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.597327    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-workload-certs" pod "363eabf7-2a32-4f2c-9333-d03592a4b41d" (UID: "363eabf7-2a32-4f2c-9333-d03592a4b41d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:53.602911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.597518    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-credential-socket" pod "363eabf7-2a32-4f2c-9333-d03592a4b41d" (UID: "363eabf7-2a32-4f2c-9333-d03592a4b41d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:53.602911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.597574    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-token\" (UniqueName: \"kubernetes.io/projected/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-token\") pod \"363eabf7-2a32-4f2c-9333-d03592a4b41d\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") "
Nov 02 17:53:53.602911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.597626    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/363eabf7-2a32-4f2c-9333-d03592a4b41d-kube-api-access-x26kd\" (UniqueName: \"kubernetes.io/projected/363eabf7-2a32-4f2c-9333-d03592a4b41d-kube-api-access-x26kd\") pod \"363eabf7-2a32-4f2c-9333-d03592a4b41d\" (UID: \"363eabf7-2a32-4f2c-9333-d03592a4b41d\") "
Nov 02 17:53:53.602911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.598574    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/363eabf7-2a32-4f2c-9333-d03592a4b41d-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.602911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.598594    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.602911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.598620    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.602911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.598633    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.602911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.598646    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.611869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.608018    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-podinfo" pod "363eabf7-2a32-4f2c-9333-d03592a4b41d" (UID: "363eabf7-2a32-4f2c-9333-d03592a4b41d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:53:53.611869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.608141    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363eabf7-2a32-4f2c-9333-d03592a4b41d-kube-api-access-x26kd" pod "363eabf7-2a32-4f2c-9333-d03592a4b41d" (UID: "363eabf7-2a32-4f2c-9333-d03592a4b41d"). InnerVolumeSpecName "kube-api-access-x26kd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:53.611869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.609596    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-envoy" pod "363eabf7-2a32-4f2c-9333-d03592a4b41d" (UID: "363eabf7-2a32-4f2c-9333-d03592a4b41d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:53.611869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.611039    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-token" pod "363eabf7-2a32-4f2c-9333-d03592a4b41d" (UID: "363eabf7-2a32-4f2c-9333-d03592a4b41d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:53.630941 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.630904    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="20a26856-e0a4-410b-978b-4c65e143a80b" path="/var/lib/kubelet/pods/20a26856-e0a4-410b-978b-4c65e143a80b/volumes"
Nov 02 17:53:53.717900 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.707958    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-x26kd\" (UniqueName: \"kubernetes.io/projected/363eabf7-2a32-4f2c-9333-d03592a4b41d-kube-api-access-x26kd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.723467 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.723429    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.723703 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.723681    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.723829 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.723799    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/363eabf7-2a32-4f2c-9333-d03592a4b41d-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:53.842011 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.841978    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"
Nov 02 17:53:53.870028 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.869995    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"]
Nov 02 17:53:53.876700 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.876670    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-k6slc"]
Nov 02 17:53:53.932441 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.930272    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/eb819cb7-0963-4c99-91c4-24ac65143c75-kube-api-access-tv85l\" (UniqueName: \"kubernetes.io/projected/eb819cb7-0963-4c99-91c4-24ac65143c75-kube-api-access-tv85l\") pod \"eb819cb7-0963-4c99-91c4-24ac65143c75\" (UID: \"eb819cb7-0963-4c99-91c4-24ac65143c75\") "
Nov 02 17:53:53.932441 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.930479    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/eb819cb7-0963-4c99-91c4-24ac65143c75-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb819cb7-0963-4c99-91c4-24ac65143c75-maas-api-xks-ca\") pod \"eb819cb7-0963-4c99-91c4-24ac65143c75\" (UID: \"eb819cb7-0963-4c99-91c4-24ac65143c75\") "
Nov 02 17:53:53.932441 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.930591    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/eb819cb7-0963-4c99-91c4-24ac65143c75-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb819cb7-0963-4c99-91c4-24ac65143c75-maas-api-openshift-ca\") pod \"eb819cb7-0963-4c99-91c4-24ac65143c75\" (UID: \"eb819cb7-0963-4c99-91c4-24ac65143c75\") "
Nov 02 17:53:53.932441 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.932031    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/eb819cb7-0963-4c99-91c4-24ac65143c75-maas-api-openshift-ca" pod "eb819cb7-0963-4c99-91c4-24ac65143c75" (UID: "eb819cb7-0963-4c99-91c4-24ac65143c75"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:53.939189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.938968    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/eb819cb7-0963-4c99-91c4-24ac65143c75-kube-api-access-tv85l" pod "eb819cb7-0963-4c99-91c4-24ac65143c75" (UID: "eb819cb7-0963-4c99-91c4-24ac65143c75"). InnerVolumeSpecName "kube-api-access-tv85l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:53.940891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:53.939742    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/eb819cb7-0963-4c99-91c4-24ac65143c75-maas-api-xks-ca" pod "eb819cb7-0963-4c99-91c4-24ac65143c75" (UID: "eb819cb7-0963-4c99-91c4-24ac65143c75"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:53:54.032231 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:54.032191    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tv85l\" (UniqueName: \"kubernetes.io/projected/eb819cb7-0963-4c99-91c4-24ac65143c75-kube-api-access-tv85l\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:54.032231 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:54.032224    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/eb819cb7-0963-4c99-91c4-24ac65143c75-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:54.032444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:54.032249    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/eb819cb7-0963-4c99-91c4-24ac65143c75-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:54.469623 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:54.469585    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:54.560541 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:54.559717    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l" event={"ID":"eb819cb7-0963-4c99-91c4-24ac65143c75","Type":"ContainerDied","Data":"07ed931c42a138643003e09c77c29459fe28ec2af21909f685596a20e1fc4ebf"}
Nov 02 17:53:54.560541 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:54.559830    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="07ed931c42a138643003e09c77c29459fe28ec2af21909f685596a20e1fc4ebf"
Nov 02 17:53:54.560541 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:54.559913    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"
Nov 02 17:53:55.619542 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:55.619504    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="363eabf7-2a32-4f2c-9333-d03592a4b41d" path="/var/lib/kubelet/pods/363eabf7-2a32-4f2c-9333-d03592a4b41d/volumes"
Nov 02 17:53:56.267324 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.267248    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:53:56.599876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.596468    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:53:56.608482 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.608447    4456 generic.go:356] "Generic (PLEG): container finished" podID="33530571-d61c-488e-be5b-54e298cfd4e4" containerID="85b38081a920ac04873b6d419f556f5ffffef86b5115b351b3b2b35a26058e83" exitCode=0
Nov 02 17:53:56.608653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.608498    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"
Nov 02 17:53:56.608653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.608553    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd" event={"ID":"33530571-d61c-488e-be5b-54e298cfd4e4","Type":"ContainerDied","Data":"85b38081a920ac04873b6d419f556f5ffffef86b5115b351b3b2b35a26058e83"}
Nov 02 17:53:56.608653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.608592    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd" event={"ID":"33530571-d61c-488e-be5b-54e298cfd4e4","Type":"ContainerDied","Data":"b6c7f4131610b4c354ecc75c2432b628f6982c0362375095c0008a5d465e3d05"}
Nov 02 17:53:56.608653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.608621    4456 scope.go:122] "RemoveContainer" containerID="85b38081a920ac04873b6d419f556f5ffffef86b5115b351b3b2b35a26058e83"
Nov 02 17:53:56.623879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.623857    4456 scope.go:122] "RemoveContainer" containerID="64e1c83b77cad53fbb7765a03a1aae028227679f40fd0cd1509bef2eb59cd3d5"
Nov 02 17:53:56.638644 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.638620    4456 scope.go:122] "RemoveContainer" containerID="85b38081a920ac04873b6d419f556f5ffffef86b5115b351b3b2b35a26058e83"
Nov 02 17:53:56.639084 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:53:56.639054    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"85b38081a920ac04873b6d419f556f5ffffef86b5115b351b3b2b35a26058e83\": container with ID starting with 85b38081a920ac04873b6d419f556f5ffffef86b5115b351b3b2b35a26058e83 not found: ID does not exist" containerID="85b38081a920ac04873b6d419f556f5ffffef86b5115b351b3b2b35a26058e83"
Nov 02 17:53:56.639197 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.639097    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"85b38081a920ac04873b6d419f556f5ffffef86b5115b351b3b2b35a26058e83"} err="failed to get container status \"85b38081a920ac04873b6d419f556f5ffffef86b5115b351b3b2b35a26058e83\": rpc error: code = NotFound desc = could not find container \"85b38081a920ac04873b6d419f556f5ffffef86b5115b351b3b2b35a26058e83\": container with ID starting with 85b38081a920ac04873b6d419f556f5ffffef86b5115b351b3b2b35a26058e83 not found: ID does not exist"
Nov 02 17:53:56.639197 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.639122    4456 scope.go:122] "RemoveContainer" containerID="64e1c83b77cad53fbb7765a03a1aae028227679f40fd0cd1509bef2eb59cd3d5"
Nov 02 17:53:56.644228 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:53:56.644168    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"64e1c83b77cad53fbb7765a03a1aae028227679f40fd0cd1509bef2eb59cd3d5\": container with ID starting with 64e1c83b77cad53fbb7765a03a1aae028227679f40fd0cd1509bef2eb59cd3d5 not found: ID does not exist" containerID="64e1c83b77cad53fbb7765a03a1aae028227679f40fd0cd1509bef2eb59cd3d5"
Nov 02 17:53:56.644228 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.644206    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"64e1c83b77cad53fbb7765a03a1aae028227679f40fd0cd1509bef2eb59cd3d5"} err="failed to get container status \"64e1c83b77cad53fbb7765a03a1aae028227679f40fd0cd1509bef2eb59cd3d5\": rpc error: code = NotFound desc = could not find container \"64e1c83b77cad53fbb7765a03a1aae028227679f40fd0cd1509bef2eb59cd3d5\": container with ID starting with 64e1c83b77cad53fbb7765a03a1aae028227679f40fd0cd1509bef2eb59cd3d5 not found: ID does not exist"
Nov 02 17:53:56.715051 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.714887    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-credential-socket\") pod \"33530571-d61c-488e-be5b-54e298cfd4e4\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") "
Nov 02 17:53:56.715051 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.714986    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-istio-envoy\") pod \"33530571-d61c-488e-be5b-54e298cfd4e4\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") "
Nov 02 17:53:56.715051 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.715034    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33530571-d61c-488e-be5b-54e298cfd4e4-kube-api-access-4lt76\" (UniqueName: \"kubernetes.io/projected/33530571-d61c-488e-be5b-54e298cfd4e4-kube-api-access-4lt76\") pod \"33530571-d61c-488e-be5b-54e298cfd4e4\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") "
Nov 02 17:53:56.716434 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.715069    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/33530571-d61c-488e-be5b-54e298cfd4e4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33530571-d61c-488e-be5b-54e298cfd4e4-istio-podinfo\") pod \"33530571-d61c-488e-be5b-54e298cfd4e4\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") "
Nov 02 17:53:56.716434 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.715106    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/33530571-d61c-488e-be5b-54e298cfd4e4-istio-token\" (UniqueName: \"kubernetes.io/projected/33530571-d61c-488e-be5b-54e298cfd4e4-istio-token\") pod \"33530571-d61c-488e-be5b-54e298cfd4e4\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") "
Nov 02 17:53:56.716434 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.715173    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/33530571-d61c-488e-be5b-54e298cfd4e4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33530571-d61c-488e-be5b-54e298cfd4e4-istiod-ca-cert\") pod \"33530571-d61c-488e-be5b-54e298cfd4e4\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") "
Nov 02 17:53:56.716434 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.715225    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-workload-socket\") pod \"33530571-d61c-488e-be5b-54e298cfd4e4\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") "
Nov 02 17:53:56.716434 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.715484    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-workload-certs\") pod \"33530571-d61c-488e-be5b-54e298cfd4e4\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") "
Nov 02 17:53:56.716434 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.715555    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-istio-data\") pod \"33530571-d61c-488e-be5b-54e298cfd4e4\" (UID: \"33530571-d61c-488e-be5b-54e298cfd4e4\") "
Nov 02 17:53:56.717150 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.716671    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-credential-socket" pod "33530571-d61c-488e-be5b-54e298cfd4e4" (UID: "33530571-d61c-488e-be5b-54e298cfd4e4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:56.717150 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.717094    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-workload-certs" pod "33530571-d61c-488e-be5b-54e298cfd4e4" (UID: "33530571-d61c-488e-be5b-54e298cfd4e4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:56.717150 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.717104    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-istio-data" pod "33530571-d61c-488e-be5b-54e298cfd4e4" (UID: "33530571-d61c-488e-be5b-54e298cfd4e4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:56.719351 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.718038    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/33530571-d61c-488e-be5b-54e298cfd4e4-istiod-ca-cert" pod "33530571-d61c-488e-be5b-54e298cfd4e4" (UID: "33530571-d61c-488e-be5b-54e298cfd4e4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:53:56.719351 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.718472    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-workload-socket" pod "33530571-d61c-488e-be5b-54e298cfd4e4" (UID: "33530571-d61c-488e-be5b-54e298cfd4e4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:56.724079 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.721171    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33530571-d61c-488e-be5b-54e298cfd4e4-istio-token" pod "33530571-d61c-488e-be5b-54e298cfd4e4" (UID: "33530571-d61c-488e-be5b-54e298cfd4e4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:56.724079 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.723610    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/33530571-d61c-488e-be5b-54e298cfd4e4-kube-api-access-4lt76" pod "33530571-d61c-488e-be5b-54e298cfd4e4" (UID: "33530571-d61c-488e-be5b-54e298cfd4e4"). InnerVolumeSpecName "kube-api-access-4lt76". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:53:56.733526 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.733464    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/33530571-d61c-488e-be5b-54e298cfd4e4-istio-podinfo" pod "33530571-d61c-488e-be5b-54e298cfd4e4" (UID: "33530571-d61c-488e-be5b-54e298cfd4e4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:53:56.744473 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.744444    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-istio-envoy" pod "33530571-d61c-488e-be5b-54e298cfd4e4" (UID: "33530571-d61c-488e-be5b-54e298cfd4e4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:53:56.817902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.817795    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:56.817902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.817837    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:56.817902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.817846    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:56.817902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.817854    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:56.817902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.817865    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4lt76\" (UniqueName: \"kubernetes.io/projected/33530571-d61c-488e-be5b-54e298cfd4e4-kube-api-access-4lt76\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:56.817902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.817874    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/33530571-d61c-488e-be5b-54e298cfd4e4-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:56.817902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.817882    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/33530571-d61c-488e-be5b-54e298cfd4e4-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:56.817902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.817890    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/33530571-d61c-488e-be5b-54e298cfd4e4-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:56.817902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.817897    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/33530571-d61c-488e-be5b-54e298cfd4e4-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:53:56.958937 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.958905    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"]
Nov 02 17:53:56.971320 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:56.971293    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-786c42-openshift-default-77dbf5f6cc-snlwd"]
Nov 02 17:53:57.619338 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:57.619305    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="33530571-d61c-488e-be5b-54e298cfd4e4" path="/var/lib/kubelet/pods/33530571-d61c-488e-be5b-54e298cfd4e4/volumes"
Nov 02 17:53:58.100649 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:53:58.100601    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:00.014054 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:00.014013    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:00.742978 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:00.742941    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:01.936532 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:01.936496    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:02.529219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.529189    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:02.634125 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.634087    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"]
Nov 02 17:54:02.635320 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.635294    4456 cpu_manager.go:414] "containerMap: removing container" podUID="363eabf7-2a32-4f2c-9333-d03592a4b41d" containerName="istio-proxy"
Nov 02 17:54:02.635420 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.635323    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="363eabf7-2a32-4f2c-9333-d03592a4b41d" containerName="istio-proxy"
Nov 02 17:54:02.635420 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.635363    4456 cpu_manager.go:414] "containerMap: removing container" podUID="20a26856-e0a4-410b-978b-4c65e143a80b" containerName="istio-proxy"
Nov 02 17:54:02.635420 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.635371    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="20a26856-e0a4-410b-978b-4c65e143a80b" containerName="istio-proxy"
Nov 02 17:54:02.635531 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.635420    4456 cpu_manager.go:414] "containerMap: removing container" podUID="eb819cb7-0963-4c99-91c4-24ac65143c75" containerName="revoke-keys"
Nov 02 17:54:02.635531 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.635429    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="eb819cb7-0963-4c99-91c4-24ac65143c75" containerName="revoke-keys"
Nov 02 17:54:02.635531 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.635443    4456 cpu_manager.go:414] "containerMap: removing container" podUID="33530571-d61c-488e-be5b-54e298cfd4e4" containerName="istio-proxy"
Nov 02 17:54:02.635531 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.635449    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33530571-d61c-488e-be5b-54e298cfd4e4" containerName="istio-proxy"
Nov 02 17:54:02.635531 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.635477    4456 cpu_manager.go:414] "containerMap: removing container" podUID="33530571-d61c-488e-be5b-54e298cfd4e4" containerName="istio-proxy"
Nov 02 17:54:02.635531 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.635484    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="33530571-d61c-488e-be5b-54e298cfd4e4" containerName="istio-proxy"
Nov 02 17:54:02.635716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.635670    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="eb819cb7-0963-4c99-91c4-24ac65143c75" containerName="revoke-keys"
Nov 02 17:54:02.635716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.635682    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="33530571-d61c-488e-be5b-54e298cfd4e4" containerName="istio-proxy"
Nov 02 17:54:02.635716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.635699    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="33530571-d61c-488e-be5b-54e298cfd4e4" containerName="istio-proxy"
Nov 02 17:54:02.635716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.635712    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="363eabf7-2a32-4f2c-9333-d03592a4b41d" containerName="istio-proxy"
Nov 02 17:54:02.635865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.635723    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="20a26856-e0a4-410b-978b-4c65e143a80b" containerName="istio-proxy"
Nov 02 17:54:02.641499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.641477    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"]
Nov 02 17:54:02.641618 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.641563    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"
Nov 02 17:54:02.677574 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.677542    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:54:02.677790 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.677666    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:54:02.688315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.687547    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:02.729463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.729410    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf45dcfc-0683-4b5c-9d23-011d36154124-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6\" (UID: \"bf45dcfc-0683-4b5c-9d23-011d36154124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"
Nov 02 17:54:02.729765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.729734    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6rc69\" (UniqueName: \"kubernetes.io/projected/bf45dcfc-0683-4b5c-9d23-011d36154124-kube-api-access-6rc69\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6\" (UID: \"bf45dcfc-0683-4b5c-9d23-011d36154124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"
Nov 02 17:54:02.730160 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.730116    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf45dcfc-0683-4b5c-9d23-011d36154124-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6\" (UID: \"bf45dcfc-0683-4b5c-9d23-011d36154124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"
Nov 02 17:54:02.832337 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.832255    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf45dcfc-0683-4b5c-9d23-011d36154124-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6\" (UID: \"bf45dcfc-0683-4b5c-9d23-011d36154124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"
Nov 02 17:54:02.832486 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.832438    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf45dcfc-0683-4b5c-9d23-011d36154124-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6\" (UID: \"bf45dcfc-0683-4b5c-9d23-011d36154124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"
Nov 02 17:54:02.832486 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.832465    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6rc69\" (UniqueName: \"kubernetes.io/projected/bf45dcfc-0683-4b5c-9d23-011d36154124-kube-api-access-6rc69\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6\" (UID: \"bf45dcfc-0683-4b5c-9d23-011d36154124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"
Nov 02 17:54:02.832947 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.832929    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf45dcfc-0683-4b5c-9d23-011d36154124-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6\" (UID: \"bf45dcfc-0683-4b5c-9d23-011d36154124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"
Nov 02 17:54:02.835072 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.835043    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf45dcfc-0683-4b5c-9d23-011d36154124-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6\" (UID: \"bf45dcfc-0683-4b5c-9d23-011d36154124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"
Nov 02 17:54:02.861212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.861180    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6rc69\" (UniqueName: \"kubernetes.io/projected/bf45dcfc-0683-4b5c-9d23-011d36154124-kube-api-access-6rc69\") pod \"maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6\" (UID: \"bf45dcfc-0683-4b5c-9d23-011d36154124\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"
Nov 02 17:54:02.989340 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:02.989311    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"
Nov 02 17:54:03.501008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:03.500977    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"]
Nov 02 17:54:03.502152 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:54:03.502128    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbf45dcfc_0683_4b5c_9d23_011d36154124.slice/crio-ba6f14ee5735605fbf5c21bfd006596aeda3c7a95c32ec612ee1ee582f3cc729 WatchSource:0}: Error finding container ba6f14ee5735605fbf5c21bfd006596aeda3c7a95c32ec612ee1ee582f3cc729: Status 404 returned error can't find the container with id ba6f14ee5735605fbf5c21bfd006596aeda3c7a95c32ec612ee1ee582f3cc729
Nov 02 17:54:03.696141 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:03.696103    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6" event={"ID":"bf45dcfc-0683-4b5c-9d23-011d36154124","Type":"ContainerStarted","Data":"ba6f14ee5735605fbf5c21bfd006596aeda3c7a95c32ec612ee1ee582f3cc729"}
Nov 02 17:54:03.937606 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:03.937548    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:04.707100 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:04.707063    4456 generic.go:356] "Generic (PLEG): container finished" podID="bf45dcfc-0683-4b5c-9d23-011d36154124" containerID="aedc3f687f79a037c431dd3d6319190e8a3019b2726ccc734c9a41664b91d46f" exitCode=0
Nov 02 17:54:04.707589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:04.707103    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6" event={"ID":"bf45dcfc-0683-4b5c-9d23-011d36154124","Type":"ContainerDied","Data":"aedc3f687f79a037c431dd3d6319190e8a3019b2726ccc734c9a41664b91d46f"}
Nov 02 17:54:05.818533 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:05.818490    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:05.947693 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:05.947659    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"
Nov 02 17:54:06.107394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:06.107310    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bf45dcfc-0683-4b5c-9d23-011d36154124-kube-api-access-6rc69\" (UniqueName: \"kubernetes.io/projected/bf45dcfc-0683-4b5c-9d23-011d36154124-kube-api-access-6rc69\") pod \"bf45dcfc-0683-4b5c-9d23-011d36154124\" (UID: \"bf45dcfc-0683-4b5c-9d23-011d36154124\") "
Nov 02 17:54:06.107394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:06.107383    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bf45dcfc-0683-4b5c-9d23-011d36154124-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf45dcfc-0683-4b5c-9d23-011d36154124-maas-api-openshift-ca\") pod \"bf45dcfc-0683-4b5c-9d23-011d36154124\" (UID: \"bf45dcfc-0683-4b5c-9d23-011d36154124\") "
Nov 02 17:54:06.107584 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:06.107430    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bf45dcfc-0683-4b5c-9d23-011d36154124-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf45dcfc-0683-4b5c-9d23-011d36154124-maas-api-xks-ca\") pod \"bf45dcfc-0683-4b5c-9d23-011d36154124\" (UID: \"bf45dcfc-0683-4b5c-9d23-011d36154124\") "
Nov 02 17:54:06.107738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:06.107713    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bf45dcfc-0683-4b5c-9d23-011d36154124-maas-api-openshift-ca" pod "bf45dcfc-0683-4b5c-9d23-011d36154124" (UID: "bf45dcfc-0683-4b5c-9d23-011d36154124"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:06.108358 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:06.108331    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bf45dcfc-0683-4b5c-9d23-011d36154124-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:06.109672 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:06.109648    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bf45dcfc-0683-4b5c-9d23-011d36154124-maas-api-xks-ca" pod "bf45dcfc-0683-4b5c-9d23-011d36154124" (UID: "bf45dcfc-0683-4b5c-9d23-011d36154124"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:54:06.109752 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:06.109734    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bf45dcfc-0683-4b5c-9d23-011d36154124-kube-api-access-6rc69" pod "bf45dcfc-0683-4b5c-9d23-011d36154124" (UID: "bf45dcfc-0683-4b5c-9d23-011d36154124"). InnerVolumeSpecName "kube-api-access-6rc69". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:06.210452 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:06.210422    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6rc69\" (UniqueName: \"kubernetes.io/projected/bf45dcfc-0683-4b5c-9d23-011d36154124-kube-api-access-6rc69\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:06.210452 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:06.210449    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bf45dcfc-0683-4b5c-9d23-011d36154124-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:06.725212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:06.725176    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"
Nov 02 17:54:06.725394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:06.725227    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6" event={"ID":"bf45dcfc-0683-4b5c-9d23-011d36154124","Type":"ContainerDied","Data":"ba6f14ee5735605fbf5c21bfd006596aeda3c7a95c32ec612ee1ee582f3cc729"}
Nov 02 17:54:06.725394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:06.725282    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ba6f14ee5735605fbf5c21bfd006596aeda3c7a95c32ec612ee1ee582f3cc729"
Nov 02 17:54:07.314871 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:07.314799    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:07.611163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:07.611064    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:07.757772 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:07.757723    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:08.263339 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:08.263304    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:09.780760 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:09.780726    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:11.630478 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:11.630424    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:13.143325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:13.143286    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"]
Nov 02 17:54:13.298652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:13.298611    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:13.452313 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:13.452280    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:15.153079 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:15.153047    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:16.174760 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:54:16.174721    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:54:16.811780 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:16.811744    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:18.144044 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.144001    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7" podUID="225f3685-fcc5-4593-a4c9-29da1e9c928d" containerName="payload-pre-processing" containerID="cri-o://a70b4f24a2d3943b27a5b5c248070d3b07a229faf1b652ddd4e244c25a62e3d0" gracePeriod=25
Nov 02 17:54:18.487294 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.487256    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"
Nov 02 17:54:18.564219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.564180    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:18.646324 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.646279    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/225f3685-fcc5-4593-a4c9-29da1e9c928d-kube-api-access-lmwsg\" (UniqueName: \"kubernetes.io/projected/225f3685-fcc5-4593-a4c9-29da1e9c928d-kube-api-access-lmwsg\") pod \"225f3685-fcc5-4593-a4c9-29da1e9c928d\" (UID: \"225f3685-fcc5-4593-a4c9-29da1e9c928d\") "
Nov 02 17:54:18.646808 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.646784    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/225f3685-fcc5-4593-a4c9-29da1e9c928d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225f3685-fcc5-4593-a4c9-29da1e9c928d-plugins-config-volume\") pod \"225f3685-fcc5-4593-a4c9-29da1e9c928d\" (UID: \"225f3685-fcc5-4593-a4c9-29da1e9c928d\") "
Nov 02 17:54:18.647249 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.647215    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/225f3685-fcc5-4593-a4c9-29da1e9c928d-plugins-config-volume" pod "225f3685-fcc5-4593-a4c9-29da1e9c928d" (UID: "225f3685-fcc5-4593-a4c9-29da1e9c928d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:18.647980 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.647955    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/225f3685-fcc5-4593-a4c9-29da1e9c928d-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:18.652205 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.652175    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/225f3685-fcc5-4593-a4c9-29da1e9c928d-kube-api-access-lmwsg" pod "225f3685-fcc5-4593-a4c9-29da1e9c928d" (UID: "225f3685-fcc5-4593-a4c9-29da1e9c928d"). InnerVolumeSpecName "kube-api-access-lmwsg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:18.750024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.749930    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lmwsg\" (UniqueName: \"kubernetes.io/projected/225f3685-fcc5-4593-a4c9-29da1e9c928d-kube-api-access-lmwsg\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:18.834583 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.834544    4456 generic.go:356] "Generic (PLEG): container finished" podID="225f3685-fcc5-4593-a4c9-29da1e9c928d" containerID="a70b4f24a2d3943b27a5b5c248070d3b07a229faf1b652ddd4e244c25a62e3d0" exitCode=0
Nov 02 17:54:18.834787 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.834596    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"
Nov 02 17:54:18.834787 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.834631    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7" event={"ID":"225f3685-fcc5-4593-a4c9-29da1e9c928d","Type":"ContainerDied","Data":"a70b4f24a2d3943b27a5b5c248070d3b07a229faf1b652ddd4e244c25a62e3d0"}
Nov 02 17:54:18.834787 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.834692    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7" event={"ID":"225f3685-fcc5-4593-a4c9-29da1e9c928d","Type":"ContainerDied","Data":"246da8bfa7288a718dd41eefbce77d391cfcdb15b3a7d925b0f3450a73d3d36a"}
Nov 02 17:54:18.834787 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.834722    4456 scope.go:122] "RemoveContainer" containerID="a70b4f24a2d3943b27a5b5c248070d3b07a229faf1b652ddd4e244c25a62e3d0"
Nov 02 17:54:18.850884 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.850864    4456 scope.go:122] "RemoveContainer" containerID="a70b4f24a2d3943b27a5b5c248070d3b07a229faf1b652ddd4e244c25a62e3d0"
Nov 02 17:54:18.851217 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:54:18.851199    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a70b4f24a2d3943b27a5b5c248070d3b07a229faf1b652ddd4e244c25a62e3d0\": container with ID starting with a70b4f24a2d3943b27a5b5c248070d3b07a229faf1b652ddd4e244c25a62e3d0 not found: ID does not exist" containerID="a70b4f24a2d3943b27a5b5c248070d3b07a229faf1b652ddd4e244c25a62e3d0"
Nov 02 17:54:18.851281 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.851224    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a70b4f24a2d3943b27a5b5c248070d3b07a229faf1b652ddd4e244c25a62e3d0"} err="failed to get container status \"a70b4f24a2d3943b27a5b5c248070d3b07a229faf1b652ddd4e244c25a62e3d0\": rpc error: code = NotFound desc = could not find container \"a70b4f24a2d3943b27a5b5c248070d3b07a229faf1b652ddd4e244c25a62e3d0\": container with ID starting with a70b4f24a2d3943b27a5b5c248070d3b07a229faf1b652ddd4e244c25a62e3d0 not found: ID does not exist"
Nov 02 17:54:18.891361 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.891328    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"]
Nov 02 17:54:18.903394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:18.903354    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-mt-04e682fe-855ff74f9f-5vzk7"]
Nov 02 17:54:19.617798 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:19.617762    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="225f3685-fcc5-4593-a4c9-29da1e9c928d" path="/var/lib/kubelet/pods/225f3685-fcc5-4593-a4c9-29da1e9c928d/volumes"
Nov 02 17:54:20.332370 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:20.332331    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:22.078537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:22.078505    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:23.892473 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:23.892429    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:25.961262 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:25.961050    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:27.529517 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:27.529479    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:27.674639 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:27.674573    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:29.431800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:29.431770    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:31.425728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:31.425688    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:33.304212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:33.304178    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:34.838938 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:34.838852    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:34.940044 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:34.940003    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:54:34.940219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:34.940085    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4" podUID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:35.308170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:35.308131    4456 generic.go:356] "Generic (PLEG): container finished" podID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" containerID="5d9cae3ca58d585af27bfd3ba6620280816bb0024b13b0622caebfcbac09cf0b" exitCode=0
Nov 02 17:54:35.308603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:35.308574    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4" event={"ID":"65156215-be49-4a1b-a7fd-2b1eb2f3f13d","Type":"ContainerDied","Data":"5d9cae3ca58d585af27bfd3ba6620280816bb0024b13b0622caebfcbac09cf0b"}
Nov 02 17:54:35.309380 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:35.309355    4456 scope.go:122] "RemoveContainer" containerID="5d9cae3ca58d585af27bfd3ba6620280816bb0024b13b0622caebfcbac09cf0b"
Nov 02 17:54:35.312956 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:35.312910    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:35.313201 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:35.313180    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:35.313340 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:35.313325    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:35.325165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:35.325135    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:35.499162 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:35.498374    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:35.644990 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:35.642557    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"]
Nov 02 17:54:35.648903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:35.648828    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-mt-04e682fe-x2rk6"]
Nov 02 17:54:35.656934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:35.656903    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:35.815263 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:35.815227    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:36.143006 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:36.142936    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:36.321236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:36.321193    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4" event={"ID":"65156215-be49-4a1b-a7fd-2b1eb2f3f13d","Type":"ContainerStarted","Data":"6e6b3298b09594bcb799769b170054e6e96452903ad9152cb54b29e190aafdb3"}
Nov 02 17:54:36.718666 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:36.718632    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:36.976427 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:36.976342    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:54:36.977564 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:36.977539    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 17:54:36.977694 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:36.977589    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4" podUID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 17:54:37.631531 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:37.631491    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bf45dcfc-0683-4b5c-9d23-011d36154124" path="/var/lib/kubelet/pods/bf45dcfc-0683-4b5c-9d23-011d36154124/volumes"
Nov 02 17:54:37.835796 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:37.835762    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:37.976974 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:37.976939    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 17:54:37.977168 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:37.976993    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4" podUID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 17:54:38.976042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:38.976008    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 17:54:38.976544 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:38.976079    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4" podUID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 17:54:39.555039 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:39.555000    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:39.976932 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:39.976901    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 17:54:39.977331 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:39.976978    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4" podUID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 17:54:40.724620 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:40.724575    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:40.993904 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:40.993667    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused" start-of-body=
Nov 02 17:54:40.993904 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:40.993743    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4" podUID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.160:15021/healthz/ready\": dial tcp 10.134.0.160:15021: connect: connection refused"
Nov 02 17:54:41.177552 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:41.177521    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:41.337710 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:41.337624    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:41.980468 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:41.980438    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:54:41.980943 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:41.980928    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:54:41.981550 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:41.981530    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 17:54:42.843991 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:42.843951    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"]
Nov 02 17:54:42.844484 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:42.844219    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb" podUID="602b86d5-0981-4f7e-8b5d-2c364e55e60e" containerName="maas-api" containerID="cri-o://0d89b1a40dda5d882a2077d0325e0db3911431a68ff9fafdd8fcb91ac3a83303" gracePeriod=30
Nov 02 17:54:42.934504 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:42.934470    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:43.331548 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.331512    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:54:43.380909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.380869    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/602b86d5-0981-4f7e-8b5d-2c364e55e60e-metrics-certs\" (UniqueName: \"kubernetes.io/secret/602b86d5-0981-4f7e-8b5d-2c364e55e60e-metrics-certs\") pod \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") "
Nov 02 17:54:43.381306 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.381288    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/602b86d5-0981-4f7e-8b5d-2c364e55e60e-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602b86d5-0981-4f7e-8b5d-2c364e55e60e-openshift-service-ca\") pod \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") "
Nov 02 17:54:43.381565 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.381550    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/602b86d5-0981-4f7e-8b5d-2c364e55e60e-kube-api-access-hwvrt\" (UniqueName: \"kubernetes.io/projected/602b86d5-0981-4f7e-8b5d-2c364e55e60e-kube-api-access-hwvrt\") pod \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") "
Nov 02 17:54:43.381888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.381866    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/602b86d5-0981-4f7e-8b5d-2c364e55e60e-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602b86d5-0981-4f7e-8b5d-2c364e55e60e-maas-api-tls\") pod \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\" (UID: \"602b86d5-0981-4f7e-8b5d-2c364e55e60e\") "
Nov 02 17:54:43.382376 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.382330    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/602b86d5-0981-4f7e-8b5d-2c364e55e60e-openshift-service-ca" pod "602b86d5-0981-4f7e-8b5d-2c364e55e60e" (UID: "602b86d5-0981-4f7e-8b5d-2c364e55e60e"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:43.382939 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.382916    4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/602b86d5-0981-4f7e-8b5d-2c364e55e60e-openshift-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:43.385600 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.385574    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/602b86d5-0981-4f7e-8b5d-2c364e55e60e-metrics-certs" pod "602b86d5-0981-4f7e-8b5d-2c364e55e60e" (UID: "602b86d5-0981-4f7e-8b5d-2c364e55e60e"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:54:43.385746 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.385725    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/602b86d5-0981-4f7e-8b5d-2c364e55e60e-kube-api-access-hwvrt" pod "602b86d5-0981-4f7e-8b5d-2c364e55e60e" (UID: "602b86d5-0981-4f7e-8b5d-2c364e55e60e"). InnerVolumeSpecName "kube-api-access-hwvrt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:43.386480 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.386457    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/602b86d5-0981-4f7e-8b5d-2c364e55e60e-maas-api-tls" pod "602b86d5-0981-4f7e-8b5d-2c364e55e60e" (UID: "602b86d5-0981-4f7e-8b5d-2c364e55e60e"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:54:43.484589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.484549    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hwvrt\" (UniqueName: \"kubernetes.io/projected/602b86d5-0981-4f7e-8b5d-2c364e55e60e-kube-api-access-hwvrt\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:43.484589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.484589    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/602b86d5-0981-4f7e-8b5d-2c364e55e60e-maas-api-tls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:43.484881 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.484603    4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/602b86d5-0981-4f7e-8b5d-2c364e55e60e-metrics-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:43.512890 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.508694    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:43.525399 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.524644    4456 generic.go:356] "Generic (PLEG): container finished" podID="602b86d5-0981-4f7e-8b5d-2c364e55e60e" containerID="0d89b1a40dda5d882a2077d0325e0db3911431a68ff9fafdd8fcb91ac3a83303" exitCode=0
Nov 02 17:54:43.525399 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.524955    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb" event={"ID":"602b86d5-0981-4f7e-8b5d-2c364e55e60e","Type":"ContainerDied","Data":"0d89b1a40dda5d882a2077d0325e0db3911431a68ff9fafdd8fcb91ac3a83303"}
Nov 02 17:54:43.525399 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.525057    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb" event={"ID":"602b86d5-0981-4f7e-8b5d-2c364e55e60e","Type":"ContainerDied","Data":"d0c052d2b5de9330922010bac93ae6beb998c6da53cc144128597d96023d29db"}
Nov 02 17:54:43.525399 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.525119    4456 scope.go:122] "RemoveContainer" containerID="0d89b1a40dda5d882a2077d0325e0db3911431a68ff9fafdd8fcb91ac3a83303"
Nov 02 17:54:43.526905 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.526257    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"
Nov 02 17:54:43.595847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.595798    4456 scope.go:122] "RemoveContainer" containerID="0d89b1a40dda5d882a2077d0325e0db3911431a68ff9fafdd8fcb91ac3a83303"
Nov 02 17:54:43.596475 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:54:43.596448    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0d89b1a40dda5d882a2077d0325e0db3911431a68ff9fafdd8fcb91ac3a83303\": container with ID starting with 0d89b1a40dda5d882a2077d0325e0db3911431a68ff9fafdd8fcb91ac3a83303 not found: ID does not exist" containerID="0d89b1a40dda5d882a2077d0325e0db3911431a68ff9fafdd8fcb91ac3a83303"
Nov 02 17:54:43.596636 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.596612    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0d89b1a40dda5d882a2077d0325e0db3911431a68ff9fafdd8fcb91ac3a83303"} err="failed to get container status \"0d89b1a40dda5d882a2077d0325e0db3911431a68ff9fafdd8fcb91ac3a83303\": rpc error: code = NotFound desc = could not find container \"0d89b1a40dda5d882a2077d0325e0db3911431a68ff9fafdd8fcb91ac3a83303\": container with ID starting with 0d89b1a40dda5d882a2077d0325e0db3911431a68ff9fafdd8fcb91ac3a83303 not found: ID does not exist"
Nov 02 17:54:43.672151 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.672023    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:43.694548 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.694513    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"]
Nov 02 17:54:43.694745 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.694561    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb"]
Nov 02 17:54:43.694745 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.694584    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"]
Nov 02 17:54:43.694745 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.694603    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-769b70-f4ccj"]
Nov 02 17:54:43.816189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:43.816087    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:44.215081 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:44.215016    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:45.645397 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:45.645315    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:45.654042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:45.654011    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="602b86d5-0981-4f7e-8b5d-2c364e55e60e" path="/var/lib/kubelet/pods/602b86d5-0981-4f7e-8b5d-2c364e55e60e/volumes"
Nov 02 17:54:45.657891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:45.654698    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e1d57a75-2039-44db-abae-0085e773803f" path="/var/lib/kubelet/pods/e1d57a75-2039-44db-abae-0085e773803f/volumes"
Nov 02 17:54:46.064281 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:46.064235    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-api-b-769b70-574c6dd9c5-4x7rb" podUID="602b86d5-0981-4f7e-8b5d-2c364e55e60e" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.149:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:54:46.174548 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:54:46.174505    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:54:46.265478 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:46.265439    4456 scope.go:122] "RemoveContainer" containerID="e753de84f69349c10975edd00c14c84deea4e4b9c17dc64bec1403582cb06593"
Nov 02 17:54:47.378448 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:47.378396    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:49.226900 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:49.226859    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:51.177419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:51.177379    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:51.356780 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:51.356736    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:51.502272 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:51.502242    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:51.640095 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:51.640060    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"]
Nov 02 17:54:51.640766 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:51.640316    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr" podUID="92a4895f-f62c-4012-8114-49a4e97a8644" containerName="istio-proxy" containerID="cri-o://28b2767f02ab7967606653f3fb185bb68f6a2fc0cd525c98bffcc4b47c14c6c0" gracePeriod=30
Nov 02 17:54:51.702960 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:51.702924    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:52.504880 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.504844    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:52.893783 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.893687    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"]
Nov 02 17:54:52.894864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.894840    4456 cpu_manager.go:414] "containerMap: removing container" podUID="bf45dcfc-0683-4b5c-9d23-011d36154124" containerName="revoke-keys"
Nov 02 17:54:52.895094 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.895078    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bf45dcfc-0683-4b5c-9d23-011d36154124" containerName="revoke-keys"
Nov 02 17:54:52.895280 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.895268    4456 cpu_manager.go:414] "containerMap: removing container" podUID="225f3685-fcc5-4593-a4c9-29da1e9c928d" containerName="payload-pre-processing"
Nov 02 17:54:52.895362 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.895351    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="225f3685-fcc5-4593-a4c9-29da1e9c928d" containerName="payload-pre-processing"
Nov 02 17:54:52.895462 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.895453    4456 cpu_manager.go:414] "containerMap: removing container" podUID="602b86d5-0981-4f7e-8b5d-2c364e55e60e" containerName="maas-api"
Nov 02 17:54:52.895546 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.895536    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="602b86d5-0981-4f7e-8b5d-2c364e55e60e" containerName="maas-api"
Nov 02 17:54:52.895837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.895807    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="bf45dcfc-0683-4b5c-9d23-011d36154124" containerName="revoke-keys"
Nov 02 17:54:52.895964 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.895954    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="225f3685-fcc5-4593-a4c9-29da1e9c928d" containerName="payload-pre-processing"
Nov 02 17:54:52.896051 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.896041    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="602b86d5-0981-4f7e-8b5d-2c364e55e60e" containerName="maas-api"
Nov 02 17:54:52.901278 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.901257    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:52.924196 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.924168    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"]
Nov 02 17:54:52.926643 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.926618    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a5c1da-openshift-default-dockercfg-gjtlc\""
Nov 02 17:54:52.980200 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.980167    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4161075-440d-4373-b878-57c77612c6d2-istio-token\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:52.980398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.980241    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4161075-440d-4373-b878-57c77612c6d2-istiod-ca-cert\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:52.980398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.980318    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-workload-certs\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:52.980398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.980386    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-istio-envoy\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:52.980577 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.980420    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-credential-socket\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:52.980577 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.980529    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-workload-socket\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:52.980686 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.980619    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-grksl\" (UniqueName: \"kubernetes.io/projected/d4161075-440d-4373-b878-57c77612c6d2-kube-api-access-grksl\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:52.980740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.980680    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-istio-data\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:52.980791 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.980753    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4161075-440d-4373-b878-57c77612c6d2-istio-podinfo\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:52.995312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:52.995286    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:53.082669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.082628    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4161075-440d-4373-b878-57c77612c6d2-istio-token\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.082893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.082716    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4161075-440d-4373-b878-57c77612c6d2-istiod-ca-cert\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.082893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.082801    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-workload-certs\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.082893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.082890    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-istio-envoy\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.083089 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.082925    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-credential-socket\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.083089 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.083033    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-workload-socket\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.083186 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.083124    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-grksl\" (UniqueName: \"kubernetes.io/projected/d4161075-440d-4373-b878-57c77612c6d2-kube-api-access-grksl\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.083253 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.083188    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-istio-data\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.083321 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.083254    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4161075-440d-4373-b878-57c77612c6d2-istio-podinfo\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.112728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.103234    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4161075-440d-4373-b878-57c77612c6d2-istiod-ca-cert\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.112728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.103869    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-workload-certs\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.112728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.104387    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-istio-data\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.112728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.104625    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-credential-socket\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.135837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.117081    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-istio-envoy\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.135837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.132926    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-workload-socket\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.162893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.159324    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4161075-440d-4373-b878-57c77612c6d2-istio-podinfo\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.162893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.161020    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:53.265912 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.265085    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4161075-440d-4373-b878-57c77612c6d2-istio-token\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.355527 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.355484    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-grksl\" (UniqueName: \"kubernetes.io/projected/d4161075-440d-4373-b878-57c77612c6d2-kube-api-access-grksl\") pod \"e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") " pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.534376 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.534347    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:53.880829 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.880779    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"]
Nov 02 17:54:53.881157 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:54:53.881124    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4161075_440d_4373_b878_57c77612c6d2.slice/crio-67a56185b766be865af25189d9f16bedea44d4a32a57cfd7c72c6d0fc82578e3 WatchSource:0}: Error finding container 67a56185b766be865af25189d9f16bedea44d4a32a57cfd7c72c6d0fc82578e3: Status 404 returned error can't find the container with id 67a56185b766be865af25189d9f16bedea44d4a32a57cfd7c72c6d0fc82578e3
Nov 02 17:54:53.884506 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.884457    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:53.884617 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.884546    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:53.884678 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:53.884613    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:54:54.674135 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:54.674077    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:54.675898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:54.675646    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" event={"ID":"d4161075-440d-4373-b878-57c77612c6d2","Type":"ContainerStarted","Data":"4a6f1801431eae7eaddf5eebcb23a321305fb236e58f980eafe4d90518d86398"}
Nov 02 17:54:54.675898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:54.675701    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" event={"ID":"d4161075-440d-4373-b878-57c77612c6d2","Type":"ContainerStarted","Data":"67a56185b766be865af25189d9f16bedea44d4a32a57cfd7c72c6d0fc82578e3"}
Nov 02 17:54:54.865978 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:54.865939    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:55.534920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:55.534889    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:55.541735 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:55.541708    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:55.606677 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:55.606613    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" podStartSLOduration=3.606595157 podStartE2EDuration="3.606595157s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:54.73642406 +0000 UTC m=+1511.227487222" watchObservedRunningTime="2026-11-02 17:54:55.606595157 +0000 UTC m=+1512.097658319"
Nov 02 17:54:55.685263 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:55.685230    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:55.686287 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:55.686266    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:54:56.156929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.156889    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"]
Nov 02 17:54:56.160767 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.160743    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"
Nov 02 17:54:56.211096 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.211057    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bxf77\" (UniqueName: \"kubernetes.io/projected/5c11216b-95b5-42ed-bc15-fe3bc02945c7-kube-api-access-bxf77\") pod \"maas-api-revoke-keys-e2e-api-a-c3c112-pw52x\" (UID: \"5c11216b-95b5-42ed-bc15-fe3bc02945c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"
Nov 02 17:54:56.211275 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.211214    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c11216b-95b5-42ed-bc15-fe3bc02945c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3c112-pw52x\" (UID: \"5c11216b-95b5-42ed-bc15-fe3bc02945c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"
Nov 02 17:54:56.211390 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.211367    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c11216b-95b5-42ed-bc15-fe3bc02945c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3c112-pw52x\" (UID: \"5c11216b-95b5-42ed-bc15-fe3bc02945c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"
Nov 02 17:54:56.212394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.212374    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:54:56.212503 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.212458    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:54:56.234797 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.234768    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"]
Nov 02 17:54:56.313252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.313213    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c11216b-95b5-42ed-bc15-fe3bc02945c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3c112-pw52x\" (UID: \"5c11216b-95b5-42ed-bc15-fe3bc02945c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"
Nov 02 17:54:56.313408 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.313367    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c11216b-95b5-42ed-bc15-fe3bc02945c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3c112-pw52x\" (UID: \"5c11216b-95b5-42ed-bc15-fe3bc02945c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"
Nov 02 17:54:56.313692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.313670    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bxf77\" (UniqueName: \"kubernetes.io/projected/5c11216b-95b5-42ed-bc15-fe3bc02945c7-kube-api-access-bxf77\") pod \"maas-api-revoke-keys-e2e-api-a-c3c112-pw52x\" (UID: \"5c11216b-95b5-42ed-bc15-fe3bc02945c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"
Nov 02 17:54:56.314122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.314094    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c11216b-95b5-42ed-bc15-fe3bc02945c7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3c112-pw52x\" (UID: \"5c11216b-95b5-42ed-bc15-fe3bc02945c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"
Nov 02 17:54:56.316075 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.316052    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c11216b-95b5-42ed-bc15-fe3bc02945c7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-a-c3c112-pw52x\" (UID: \"5c11216b-95b5-42ed-bc15-fe3bc02945c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"
Nov 02 17:54:56.341851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.341790    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bxf77\" (UniqueName: \"kubernetes.io/projected/5c11216b-95b5-42ed-bc15-fe3bc02945c7-kube-api-access-bxf77\") pod \"maas-api-revoke-keys-e2e-api-a-c3c112-pw52x\" (UID: \"5c11216b-95b5-42ed-bc15-fe3bc02945c7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"
Nov 02 17:54:56.475343 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.475306    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"
Nov 02 17:54:56.815194 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.815160    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"]
Nov 02 17:54:56.815611 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:54:56.815551    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5c11216b_95b5_42ed_bc15_fe3bc02945c7.slice/crio-0f8d4d78c00324700573b15b034d970e64bc95cb94f1c3e715c21815eb601fc9 WatchSource:0}: Error finding container 0f8d4d78c00324700573b15b034d970e64bc95cb94f1c3e715c21815eb601fc9: Status 404 returned error can't find the container with id 0f8d4d78c00324700573b15b034d970e64bc95cb94f1c3e715c21815eb601fc9
Nov 02 17:54:56.845725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.842347    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:56.979163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:56.979137    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:54:57.049956 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.049884    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-credential-socket\") pod \"92a4895f-f62c-4012-8114-49a4e97a8644\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") "
Nov 02 17:54:57.049956 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.049936    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a4895f-f62c-4012-8114-49a4e97a8644-istio-token\" (UniqueName: \"kubernetes.io/projected/92a4895f-f62c-4012-8114-49a4e97a8644-istio-token\") pod \"92a4895f-f62c-4012-8114-49a4e97a8644\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") "
Nov 02 17:54:57.050141 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.049979    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-istio-data\") pod \"92a4895f-f62c-4012-8114-49a4e97a8644\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") "
Nov 02 17:54:57.050141 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.050046    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92a4895f-f62c-4012-8114-49a4e97a8644-kube-api-access-pfb5r\" (UniqueName: \"kubernetes.io/projected/92a4895f-f62c-4012-8114-49a4e97a8644-kube-api-access-pfb5r\") pod \"92a4895f-f62c-4012-8114-49a4e97a8644\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") "
Nov 02 17:54:57.050244 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.050163    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-workload-socket\") pod \"92a4895f-f62c-4012-8114-49a4e97a8644\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") "
Nov 02 17:54:57.050244 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.050195    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-istio-envoy\") pod \"92a4895f-f62c-4012-8114-49a4e97a8644\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") "
Nov 02 17:54:57.050244 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.050230    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-credential-socket" pod "92a4895f-f62c-4012-8114-49a4e97a8644" (UID: "92a4895f-f62c-4012-8114-49a4e97a8644"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:57.050370 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.050252    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92a4895f-f62c-4012-8114-49a4e97a8644-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a4895f-f62c-4012-8114-49a4e97a8644-istio-podinfo\") pod \"92a4895f-f62c-4012-8114-49a4e97a8644\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") "
Nov 02 17:54:57.050418 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.050406    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-workload-certs\") pod \"92a4895f-f62c-4012-8114-49a4e97a8644\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") "
Nov 02 17:54:57.050480 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.050407    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-istio-data" pod "92a4895f-f62c-4012-8114-49a4e97a8644" (UID: "92a4895f-f62c-4012-8114-49a4e97a8644"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:57.050567 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.050546    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92a4895f-f62c-4012-8114-49a4e97a8644-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a4895f-f62c-4012-8114-49a4e97a8644-istiod-ca-cert\") pod \"92a4895f-f62c-4012-8114-49a4e97a8644\" (UID: \"92a4895f-f62c-4012-8114-49a4e97a8644\") "
Nov 02 17:54:57.050653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.050623    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-workload-socket" pod "92a4895f-f62c-4012-8114-49a4e97a8644" (UID: "92a4895f-f62c-4012-8114-49a4e97a8644"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:57.050715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.050654    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-workload-certs" pod "92a4895f-f62c-4012-8114-49a4e97a8644" (UID: "92a4895f-f62c-4012-8114-49a4e97a8644"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:57.051030 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.050980    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92a4895f-f62c-4012-8114-49a4e97a8644-istiod-ca-cert" pod "92a4895f-f62c-4012-8114-49a4e97a8644" (UID: "92a4895f-f62c-4012-8114-49a4e97a8644"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:54:57.051665 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.051640    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:57.051665 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.051671    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:57.051879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.051688    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92a4895f-f62c-4012-8114-49a4e97a8644-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:57.051879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.051701    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:57.051879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.051713    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:57.053304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.053276    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-istio-envoy" pod "92a4895f-f62c-4012-8114-49a4e97a8644" (UID: "92a4895f-f62c-4012-8114-49a4e97a8644"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:54:57.053618 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.053590    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a4895f-f62c-4012-8114-49a4e97a8644-kube-api-access-pfb5r" pod "92a4895f-f62c-4012-8114-49a4e97a8644" (UID: "92a4895f-f62c-4012-8114-49a4e97a8644"). InnerVolumeSpecName "kube-api-access-pfb5r". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:57.053789 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.053766    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92a4895f-f62c-4012-8114-49a4e97a8644-istio-podinfo" pod "92a4895f-f62c-4012-8114-49a4e97a8644" (UID: "92a4895f-f62c-4012-8114-49a4e97a8644"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:54:57.054026 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.054004    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92a4895f-f62c-4012-8114-49a4e97a8644-istio-token" pod "92a4895f-f62c-4012-8114-49a4e97a8644" (UID: "92a4895f-f62c-4012-8114-49a4e97a8644"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:54:57.153317 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.153285    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92a4895f-f62c-4012-8114-49a4e97a8644-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:57.153317 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.153316    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92a4895f-f62c-4012-8114-49a4e97a8644-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:57.153585 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.153331    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pfb5r\" (UniqueName: \"kubernetes.io/projected/92a4895f-f62c-4012-8114-49a4e97a8644-kube-api-access-pfb5r\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:57.153585 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.153345    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92a4895f-f62c-4012-8114-49a4e97a8644-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:54:57.212127 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.212073    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:57.382915 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.382879    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:57.704653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.704596    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x" event={"ID":"5c11216b-95b5-42ed-bc15-fe3bc02945c7","Type":"ContainerStarted","Data":"959c9a72d603d65a409a8b2c5aafe12f1e188fd50b942870327fa4707b5b135b"}
Nov 02 17:54:57.704843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.704683    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x" event={"ID":"5c11216b-95b5-42ed-bc15-fe3bc02945c7","Type":"ContainerStarted","Data":"0f8d4d78c00324700573b15b034d970e64bc95cb94f1c3e715c21815eb601fc9"}
Nov 02 17:54:57.707738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.707709    4456 generic.go:356] "Generic (PLEG): container finished" podID="92a4895f-f62c-4012-8114-49a4e97a8644" containerID="28b2767f02ab7967606653f3fb185bb68f6a2fc0cd525c98bffcc4b47c14c6c0" exitCode=0
Nov 02 17:54:57.707906 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.707772    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"
Nov 02 17:54:57.707906 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.707787    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr" event={"ID":"92a4895f-f62c-4012-8114-49a4e97a8644","Type":"ContainerDied","Data":"28b2767f02ab7967606653f3fb185bb68f6a2fc0cd525c98bffcc4b47c14c6c0"}
Nov 02 17:54:57.707906 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.707861    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr" event={"ID":"92a4895f-f62c-4012-8114-49a4e97a8644","Type":"ContainerDied","Data":"0145904e38a853dddb60f8d9842215f9d906cf838e03a26d5a9585358a0e2d85"}
Nov 02 17:54:57.707906 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.707884    4456 scope.go:122] "RemoveContainer" containerID="28b2767f02ab7967606653f3fb185bb68f6a2fc0cd525c98bffcc4b47c14c6c0"
Nov 02 17:54:57.721628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.721610    4456 scope.go:122] "RemoveContainer" containerID="573cf1c3514bb6d994f4e99ee4466848b20f08591ecb7d4646a422ba5b8684aa"
Nov 02 17:54:57.731846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.731826    4456 scope.go:122] "RemoveContainer" containerID="28b2767f02ab7967606653f3fb185bb68f6a2fc0cd525c98bffcc4b47c14c6c0"
Nov 02 17:54:57.732113 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:54:57.732091    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"28b2767f02ab7967606653f3fb185bb68f6a2fc0cd525c98bffcc4b47c14c6c0\": container with ID starting with 28b2767f02ab7967606653f3fb185bb68f6a2fc0cd525c98bffcc4b47c14c6c0 not found: ID does not exist" containerID="28b2767f02ab7967606653f3fb185bb68f6a2fc0cd525c98bffcc4b47c14c6c0"
Nov 02 17:54:57.732235 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.732123    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"28b2767f02ab7967606653f3fb185bb68f6a2fc0cd525c98bffcc4b47c14c6c0"} err="failed to get container status \"28b2767f02ab7967606653f3fb185bb68f6a2fc0cd525c98bffcc4b47c14c6c0\": rpc error: code = NotFound desc = could not find container \"28b2767f02ab7967606653f3fb185bb68f6a2fc0cd525c98bffcc4b47c14c6c0\": container with ID starting with 28b2767f02ab7967606653f3fb185bb68f6a2fc0cd525c98bffcc4b47c14c6c0 not found: ID does not exist"
Nov 02 17:54:57.732235 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.732147    4456 scope.go:122] "RemoveContainer" containerID="573cf1c3514bb6d994f4e99ee4466848b20f08591ecb7d4646a422ba5b8684aa"
Nov 02 17:54:57.732453 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:54:57.732432    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"573cf1c3514bb6d994f4e99ee4466848b20f08591ecb7d4646a422ba5b8684aa\": container with ID starting with 573cf1c3514bb6d994f4e99ee4466848b20f08591ecb7d4646a422ba5b8684aa not found: ID does not exist" containerID="573cf1c3514bb6d994f4e99ee4466848b20f08591ecb7d4646a422ba5b8684aa"
Nov 02 17:54:57.732495 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.732459    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"573cf1c3514bb6d994f4e99ee4466848b20f08591ecb7d4646a422ba5b8684aa"} err="failed to get container status \"573cf1c3514bb6d994f4e99ee4466848b20f08591ecb7d4646a422ba5b8684aa\": rpc error: code = NotFound desc = could not find container \"573cf1c3514bb6d994f4e99ee4466848b20f08591ecb7d4646a422ba5b8684aa\": container with ID starting with 573cf1c3514bb6d994f4e99ee4466848b20f08591ecb7d4646a422ba5b8684aa not found: ID does not exist"
Nov 02 17:54:57.745896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.745854    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x" podStartSLOduration=1.745841808 podStartE2EDuration="1.745841808s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:54:57.745390393 +0000 UTC m=+1514.236453574" watchObservedRunningTime="2026-11-02 17:54:57.745841808 +0000 UTC m=+1514.236904966"
Nov 02 17:54:57.786563 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.786539    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"]
Nov 02 17:54:57.798469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:57.798429    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-769b70-openshift-default-897676bcc-gfczr"]
Nov 02 17:54:58.721236 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:58.721202    4456 generic.go:356] "Generic (PLEG): container finished" podID="5c11216b-95b5-42ed-bc15-fe3bc02945c7" containerID="959c9a72d603d65a409a8b2c5aafe12f1e188fd50b942870327fa4707b5b135b" exitCode=0
Nov 02 17:54:58.721682 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:58.721292    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x" event={"ID":"5c11216b-95b5-42ed-bc15-fe3bc02945c7","Type":"ContainerDied","Data":"959c9a72d603d65a409a8b2c5aafe12f1e188fd50b942870327fa4707b5b135b"}
Nov 02 17:54:58.807340 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:58.807303    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:58.960484 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:58.960451    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:59.097514 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.097424    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"]
Nov 02 17:54:59.098692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.098666    4456 cpu_manager.go:414] "containerMap: removing container" podUID="92a4895f-f62c-4012-8114-49a4e97a8644" containerName="istio-proxy"
Nov 02 17:54:59.098692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.098691    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a4895f-f62c-4012-8114-49a4e97a8644" containerName="istio-proxy"
Nov 02 17:54:59.098929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.098742    4456 cpu_manager.go:414] "containerMap: removing container" podUID="92a4895f-f62c-4012-8114-49a4e97a8644" containerName="istio-proxy"
Nov 02 17:54:59.098929 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.098750    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92a4895f-f62c-4012-8114-49a4e97a8644" containerName="istio-proxy"
Nov 02 17:54:59.099010 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.098973    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a4895f-f62c-4012-8114-49a4e97a8644" containerName="istio-proxy"
Nov 02 17:54:59.102738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.102715    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 17:54:59.120191 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.119989    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a5c1da\""
Nov 02 17:54:59.120191 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.120134    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a5c1da-dockercfg-vlnwc\""
Nov 02 17:54:59.120713 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.120688    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:54:59.121681 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.121653    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"]
Nov 02 17:54:59.249847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.203015    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nklr4\" (UniqueName: \"kubernetes.io/projected/96db7b02-b878-435a-869e-1d30e147c61d-kube-api-access-nklr4\") pod \"payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg\" (UID: \"96db7b02-b878-435a-869e-1d30e147c61d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 17:54:59.250405 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.250370    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg\" (UID: \"96db7b02-b878-435a-869e-1d30e147c61d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 17:54:59.250878 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.214211    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"]
Nov 02 17:54:59.257837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.252358    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="92a4895f-f62c-4012-8114-49a4e97a8644" containerName="istio-proxy"
Nov 02 17:54:59.274840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.273872    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 17:54:59.334841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.325383    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"]
Nov 02 17:54:59.389893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.382202    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89\" (UID: \"8c60dfaa-dabe-4726-91b5-165443ff3971\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 17:54:59.389893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.382390    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qr8hh\" (UniqueName: \"kubernetes.io/projected/8c60dfaa-dabe-4726-91b5-165443ff3971-kube-api-access-qr8hh\") pod \"payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89\" (UID: \"8c60dfaa-dabe-4726-91b5-165443ff3971\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 17:54:59.389893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.382519    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nklr4\" (UniqueName: \"kubernetes.io/projected/96db7b02-b878-435a-869e-1d30e147c61d-kube-api-access-nklr4\") pod \"payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg\" (UID: \"96db7b02-b878-435a-869e-1d30e147c61d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 17:54:59.389893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.382652    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg\" (UID: \"96db7b02-b878-435a-869e-1d30e147c61d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 17:54:59.389893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.383675    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg\" (UID: \"96db7b02-b878-435a-869e-1d30e147c61d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 17:54:59.470846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.470039    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nklr4\" (UniqueName: \"kubernetes.io/projected/96db7b02-b878-435a-869e-1d30e147c61d-kube-api-access-nklr4\") pod \"payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg\" (UID: \"96db7b02-b878-435a-869e-1d30e147c61d\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 17:54:59.490865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.485725    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89\" (UID: \"8c60dfaa-dabe-4726-91b5-165443ff3971\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 17:54:59.490865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.485922    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qr8hh\" (UniqueName: \"kubernetes.io/projected/8c60dfaa-dabe-4726-91b5-165443ff3971-kube-api-access-qr8hh\") pod \"payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89\" (UID: \"8c60dfaa-dabe-4726-91b5-165443ff3971\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 17:54:59.519894 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.498385    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89\" (UID: \"8c60dfaa-dabe-4726-91b5-165443ff3971\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 17:54:59.546909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.546877    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qr8hh\" (UniqueName: \"kubernetes.io/projected/8c60dfaa-dabe-4726-91b5-165443ff3971-kube-api-access-qr8hh\") pod \"payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89\" (UID: \"8c60dfaa-dabe-4726-91b5-165443ff3971\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 17:54:59.621799 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.621766    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92a4895f-f62c-4012-8114-49a4e97a8644" path="/var/lib/kubelet/pods/92a4895f-f62c-4012-8114-49a4e97a8644/volumes"
Nov 02 17:54:59.636878 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.636850    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 17:54:59.738831 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.738795    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 17:54:59.942007 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.941980    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"
Nov 02 17:54:59.954111 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:54:59.954080    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"]
Nov 02 17:54:59.954668 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:54:59.954640    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8c60dfaa_dabe_4726_91b5_165443ff3971.slice/crio-5f90ce0fb09944493d00fc7e0feface8077779a655da9071d0560154b6550598 WatchSource:0}: Error finding container 5f90ce0fb09944493d00fc7e0feface8077779a655da9071d0560154b6550598: Status 404 returned error can't find the container with id 5f90ce0fb09944493d00fc7e0feface8077779a655da9071d0560154b6550598
Nov 02 17:55:00.090084 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.090047    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"]
Nov 02 17:55:00.094369 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:55:00.094338    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod96db7b02_b878_435a_869e_1d30e147c61d.slice/crio-cdc808bf198ef177770722cadcded05a71fa080190c7dd200fb6f2ee77c03ba2 WatchSource:0}: Error finding container cdc808bf198ef177770722cadcded05a71fa080190c7dd200fb6f2ee77c03ba2: Status 404 returned error can't find the container with id cdc808bf198ef177770722cadcded05a71fa080190c7dd200fb6f2ee77c03ba2
Nov 02 17:55:00.096951 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.096926    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5c11216b-95b5-42ed-bc15-fe3bc02945c7-kube-api-access-bxf77\" (UniqueName: \"kubernetes.io/projected/5c11216b-95b5-42ed-bc15-fe3bc02945c7-kube-api-access-bxf77\") pod \"5c11216b-95b5-42ed-bc15-fe3bc02945c7\" (UID: \"5c11216b-95b5-42ed-bc15-fe3bc02945c7\") "
Nov 02 17:55:00.097108 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.097094    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5c11216b-95b5-42ed-bc15-fe3bc02945c7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c11216b-95b5-42ed-bc15-fe3bc02945c7-maas-api-openshift-ca\") pod \"5c11216b-95b5-42ed-bc15-fe3bc02945c7\" (UID: \"5c11216b-95b5-42ed-bc15-fe3bc02945c7\") "
Nov 02 17:55:00.097155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.097140    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5c11216b-95b5-42ed-bc15-fe3bc02945c7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c11216b-95b5-42ed-bc15-fe3bc02945c7-maas-api-xks-ca\") pod \"5c11216b-95b5-42ed-bc15-fe3bc02945c7\" (UID: \"5c11216b-95b5-42ed-bc15-fe3bc02945c7\") "
Nov 02 17:55:00.097475 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.097449    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5c11216b-95b5-42ed-bc15-fe3bc02945c7-maas-api-openshift-ca" pod "5c11216b-95b5-42ed-bc15-fe3bc02945c7" (UID: "5c11216b-95b5-42ed-bc15-fe3bc02945c7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:00.098108 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.098089    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/5c11216b-95b5-42ed-bc15-fe3bc02945c7-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:00.099615 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.099588    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5c11216b-95b5-42ed-bc15-fe3bc02945c7-kube-api-access-bxf77" pod "5c11216b-95b5-42ed-bc15-fe3bc02945c7" (UID: "5c11216b-95b5-42ed-bc15-fe3bc02945c7"). InnerVolumeSpecName "kube-api-access-bxf77". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:00.100069 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.100045    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5c11216b-95b5-42ed-bc15-fe3bc02945c7-maas-api-xks-ca" pod "5c11216b-95b5-42ed-bc15-fe3bc02945c7" (UID: "5c11216b-95b5-42ed-bc15-fe3bc02945c7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:55:00.199931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.199901    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bxf77\" (UniqueName: \"kubernetes.io/projected/5c11216b-95b5-42ed-bc15-fe3bc02945c7-kube-api-access-bxf77\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:00.199931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.199932    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/5c11216b-95b5-42ed-bc15-fe3bc02945c7-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:00.638119 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.638089    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:00.746352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.746184    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x" event={"ID":"5c11216b-95b5-42ed-bc15-fe3bc02945c7","Type":"ContainerDied","Data":"0f8d4d78c00324700573b15b034d970e64bc95cb94f1c3e715c21815eb601fc9"}
Nov 02 17:55:00.746352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.746259    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0f8d4d78c00324700573b15b034d970e64bc95cb94f1c3e715c21815eb601fc9"
Nov 02 17:55:00.747071 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.746366    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"
Nov 02 17:55:00.748576 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.748550    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" event={"ID":"96db7b02-b878-435a-869e-1d30e147c61d","Type":"ContainerStarted","Data":"1780545d23b0db3b73d9b9c93d4463de2ecb228acbbac446492792422b835d1c"}
Nov 02 17:55:00.748712 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.748603    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" event={"ID":"96db7b02-b878-435a-869e-1d30e147c61d","Type":"ContainerStarted","Data":"cdc808bf198ef177770722cadcded05a71fa080190c7dd200fb6f2ee77c03ba2"}
Nov 02 17:55:00.748712 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.748637    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 17:55:00.750770 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.750738    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" event={"ID":"8c60dfaa-dabe-4726-91b5-165443ff3971","Type":"ContainerStarted","Data":"47ce80eead8cafb5ad533cbe8d61db652959180c5d1b75fef0c82fd03c99ce2f"}
Nov 02 17:55:00.750910 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.750795    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" event={"ID":"8c60dfaa-dabe-4726-91b5-165443ff3971","Type":"ContainerStarted","Data":"5f90ce0fb09944493d00fc7e0feface8077779a655da9071d0560154b6550598"}
Nov 02 17:55:00.751003 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.750967    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 17:55:00.805762 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.805701    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" podStartSLOduration=1.8056850880000002 podStartE2EDuration="1.805685088s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:00.803906071 +0000 UTC m=+1517.294969232" watchObservedRunningTime="2026-11-02 17:55:00.805685088 +0000 UTC m=+1517.296748252"
Nov 02 17:55:00.867492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:00.867438    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" podStartSLOduration=1.867422859 podStartE2EDuration="1.867422859s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:54:59 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:00.861955438 +0000 UTC m=+1517.353018621" watchObservedRunningTime="2026-11-02 17:55:00.867422859 +0000 UTC m=+1517.358486020"
Nov 02 17:55:02.557029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:02.556948    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:04.447104 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:04.447058    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:06.369219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:06.369176    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:06.881459 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:06.881426    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:07.029335 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:07.028987    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:08.132398 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:08.132360    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:08.275659 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:08.275609    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:08.821256 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:08.821201    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:10.219651 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:10.219291    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:10.361542 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:10.361403    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:11.762979 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:11.762946    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 17:55:11.763458 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:11.763028    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 17:55:11.973290 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:11.973239    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:12.511337 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:12.511295    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:12.666282 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:12.664011    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:12.808859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:12.808743    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:13.014983 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:13.014947    4456 patch_prober.go:29] interesting pod/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:55:13.015172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:13.015017    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:55:13.162353 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:13.162148    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:14.556979 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:14.556945    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:14.723148 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:14.723118    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:14.985846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:14.985797    4456 generic.go:356] "Generic (PLEG): container finished" podID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerID="8a129e0cab5c942205679f6b457e66733b2f8b0cc96304573dab32c95c14d867" exitCode=0
Nov 02 17:55:14.986013 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:14.985873    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" event={"ID":"0a41f8f2-1497-476f-b391-8b24c9c7f7f4","Type":"ContainerDied","Data":"8a129e0cab5c942205679f6b457e66733b2f8b0cc96304573dab32c95c14d867"}
Nov 02 17:55:14.986339 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:14.986323    4456 scope.go:122] "RemoveContainer" containerID="8a129e0cab5c942205679f6b457e66733b2f8b0cc96304573dab32c95c14d867"
Nov 02 17:55:14.988206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:14.988182    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:14.988262 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:14.988250    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:14.988301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:14.988285    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:15.053700 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:15.053673    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:15.681852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:15.679480    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:15.886842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:15.875367    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:16.099299 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:16.099265    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" event={"ID":"0a41f8f2-1497-476f-b391-8b24c9c7f7f4","Type":"ContainerStarted","Data":"54ec2269cb5bd8ac76efbd725adbaeb88ab8284252404474208d847effc04e1b"}
Nov 02 17:55:16.224842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:16.217244    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:55:16.224842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:16.217358    4456 patch_prober.go:29] interesting pod/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 02 17:55:16.224842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:16.217397    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 02 17:55:16.228698 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:55:16.228654    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:55:16.699840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:16.699245    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:17.211163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:17.211131    4456 patch_prober.go:29] interesting pod/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 02 17:55:17.211391 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:17.211195    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 02 17:55:18.211341 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:18.211294    4456 patch_prober.go:29] interesting pod/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 02 17:55:18.211792 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:18.211382    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 02 17:55:18.971440 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:18.971403    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:19.206840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:19.206408    4456 patch_prober.go:29] interesting pod/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 02 17:55:19.206840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:19.206476    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 02 17:55:19.834781 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:19.834741    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:20.126772 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:20.126680    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:20.209841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:20.206259    4456 patch_prober.go:29] interesting pod/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 02 17:55:20.222839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:20.210091    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 02 17:55:20.223067 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:20.208121    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"]
Nov 02 17:55:20.223285 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:20.223257    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6" podUID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" containerName="istio-proxy" containerID="cri-o://b8e26c4e4edaaffcb5446aa0eb600b76142fc6e40a41d43c05253afd1e946b94" gracePeriod=30
Nov 02 17:55:20.380241 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:20.379397    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:21.206262 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:21.206219    4456 patch_prober.go:29] interesting pod/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused" start-of-body=
Nov 02 17:55:21.206838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:21.206289    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.164:15021/healthz/ready\": dial tcp 10.134.0.164:15021: connect: connection refused"
Nov 02 17:55:21.295613 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:21.295566    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:21.847940 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:21.847901    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:22.087928 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.087797    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:22.090962 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.090933    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"]
Nov 02 17:55:22.092516 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.092457    4456 cpu_manager.go:414] "containerMap: removing container" podUID="5c11216b-95b5-42ed-bc15-fe3bc02945c7" containerName="revoke-keys"
Nov 02 17:55:22.092950 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.092912    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5c11216b-95b5-42ed-bc15-fe3bc02945c7" containerName="revoke-keys"
Nov 02 17:55:22.093350 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.093331    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5c11216b-95b5-42ed-bc15-fe3bc02945c7" containerName="revoke-keys"
Nov 02 17:55:22.164732 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.164645    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"]
Nov 02 17:55:22.165270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.164753    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"
Nov 02 17:55:22.189852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.189601    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8a209f\""
Nov 02 17:55:22.194133 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.194066    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8a209f-dockercfg-g8ztf\""
Nov 02 17:55:22.259024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.258954    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:55:22.259617 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.259592    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:55:22.260991 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.260962    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:55:22.300764 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.300726    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6phtq\" (UniqueName: \"kubernetes.io/projected/c0ff0cbb-4acf-4d63-9579-6aeac659b2ea-kube-api-access-6phtq\") pod \"payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g\" (UID: \"c0ff0cbb-4acf-4d63-9579-6aeac659b2ea\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"
Nov 02 17:55:22.301149 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.301126    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ff0cbb-4acf-4d63-9579-6aeac659b2ea-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g\" (UID: \"c0ff0cbb-4acf-4d63-9579-6aeac659b2ea\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"
Nov 02 17:55:22.450497 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.450456    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ff0cbb-4acf-4d63-9579-6aeac659b2ea-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g\" (UID: \"c0ff0cbb-4acf-4d63-9579-6aeac659b2ea\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"
Nov 02 17:55:22.450780 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.450759    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6phtq\" (UniqueName: \"kubernetes.io/projected/c0ff0cbb-4acf-4d63-9579-6aeac659b2ea-kube-api-access-6phtq\") pod \"payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g\" (UID: \"c0ff0cbb-4acf-4d63-9579-6aeac659b2ea\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"
Nov 02 17:55:22.460878 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.460544    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ff0cbb-4acf-4d63-9579-6aeac659b2ea-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g\" (UID: \"c0ff0cbb-4acf-4d63-9579-6aeac659b2ea\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"
Nov 02 17:55:22.491585 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.491507    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6phtq\" (UniqueName: \"kubernetes.io/projected/c0ff0cbb-4acf-4d63-9579-6aeac659b2ea-kube-api-access-6phtq\") pod \"payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g\" (UID: \"c0ff0cbb-4acf-4d63-9579-6aeac659b2ea\") " pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"
Nov 02 17:55:22.537391 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.537362    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"
Nov 02 17:55:22.876243 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:22.876211    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"]
Nov 02 17:55:22.876496 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:55:22.876471    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc0ff0cbb_4acf_4d63_9579_6aeac659b2ea.slice/crio-c31100af0a57f3e463e2a59f7b244d7da03e818a6765fc5a0fa944ebc056a2e8 WatchSource:0}: Error finding container c31100af0a57f3e463e2a59f7b244d7da03e818a6765fc5a0fa944ebc056a2e8: Status 404 returned error can't find the container with id c31100af0a57f3e463e2a59f7b244d7da03e818a6765fc5a0fa944ebc056a2e8
Nov 02 17:55:23.179325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:23.179225    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" event={"ID":"c0ff0cbb-4acf-4d63-9579-6aeac659b2ea","Type":"ContainerStarted","Data":"ab3b1839843faf727063beb731cd4336c7b223cdec36d36195c04bd9689f1da3"}
Nov 02 17:55:23.179325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:23.179293    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" event={"ID":"c0ff0cbb-4acf-4d63-9579-6aeac659b2ea","Type":"ContainerStarted","Data":"c31100af0a57f3e463e2a59f7b244d7da03e818a6765fc5a0fa944ebc056a2e8"}
Nov 02 17:55:23.179325 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:23.179322    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"
Nov 02 17:55:23.225517 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:23.225447    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" podStartSLOduration=1.225429335 podStartE2EDuration="1.225429335s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:23.224959183 +0000 UTC m=+1539.716022356" watchObservedRunningTime="2026-11-02 17:55:23.225429335 +0000 UTC m=+1539.716492498"
Nov 02 17:55:23.338193 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:23.338156    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:23.680837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:23.670628    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:23.964528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:23.964482    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:24.005421 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:24.005374    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"]
Nov 02 17:55:24.005919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:24.005888    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq" podUID="e37acee8-ead8-40d3-8d09-160e87271744" containerName="istio-proxy" containerID="cri-o://edc469400bd64078521d131783582abbfa00e459114777a8e5f0ea23e411cb08" gracePeriod=30
Nov 02 17:55:24.133787 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:24.133753    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:24.836311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:24.836279    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:25.324420 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.324381    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:25.476294 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.476220    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:25.806791 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.806763    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:55:25.889887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.880693    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-workload-certs\") pod \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") "
Nov 02 17:55:25.889887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.880774    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-kube-api-access-ztztx\" (UniqueName: \"kubernetes.io/projected/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-kube-api-access-ztztx\") pod \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") "
Nov 02 17:55:25.889887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.880918    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-podinfo\") pod \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") "
Nov 02 17:55:25.889887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.881068    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-credential-socket\") pod \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") "
Nov 02 17:55:25.889887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.881142    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istiod-ca-cert\") pod \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") "
Nov 02 17:55:25.889887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.881198    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-token\" (UniqueName: \"kubernetes.io/projected/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-token\") pod \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") "
Nov 02 17:55:25.889887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.881235    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-envoy\") pod \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") "
Nov 02 17:55:25.889887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.881364    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-workload-socket\") pod \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") "
Nov 02 17:55:25.889887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.881391    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-data\") pod \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\" (UID: \"0e46fada-1bd0-4b22-96ab-3c49c4b3199e\") "
Nov 02 17:55:25.889887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.882965    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-data" pod "0e46fada-1bd0-4b22-96ab-3c49c4b3199e" (UID: "0e46fada-1bd0-4b22-96ab-3c49c4b3199e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:25.905441 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.894143    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istiod-ca-cert" pod "0e46fada-1bd0-4b22-96ab-3c49c4b3199e" (UID: "0e46fada-1bd0-4b22-96ab-3c49c4b3199e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:25.905441 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.894427    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-workload-socket" pod "0e46fada-1bd0-4b22-96ab-3c49c4b3199e" (UID: "0e46fada-1bd0-4b22-96ab-3c49c4b3199e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:25.905441 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.899766    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-workload-certs" pod "0e46fada-1bd0-4b22-96ab-3c49c4b3199e" (UID: "0e46fada-1bd0-4b22-96ab-3c49c4b3199e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:25.906702 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.906624    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:25.920853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.913916    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-credential-socket" pod "0e46fada-1bd0-4b22-96ab-3c49c4b3199e" (UID: "0e46fada-1bd0-4b22-96ab-3c49c4b3199e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:25.948591 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.948441    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-kube-api-access-ztztx" pod "0e46fada-1bd0-4b22-96ab-3c49c4b3199e" (UID: "0e46fada-1bd0-4b22-96ab-3c49c4b3199e"). InnerVolumeSpecName "kube-api-access-ztztx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:25.957520 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.957483    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-token" pod "0e46fada-1bd0-4b22-96ab-3c49c4b3199e" (UID: "0e46fada-1bd0-4b22-96ab-3c49c4b3199e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:25.957850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.957615    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-envoy" pod "0e46fada-1bd0-4b22-96ab-3c49c4b3199e" (UID: "0e46fada-1bd0-4b22-96ab-3c49c4b3199e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:25.960605 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.960571    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-podinfo" pod "0e46fada-1bd0-4b22-96ab-3c49c4b3199e" (UID: "0e46fada-1bd0-4b22-96ab-3c49c4b3199e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:55:25.984112 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.984069    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:25.984112 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.984108    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ztztx\" (UniqueName: \"kubernetes.io/projected/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-kube-api-access-ztztx\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:25.984349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.984125    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:25.984349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.984138    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:25.984349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.984152    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:25.984349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.984163    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:25.984349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.984175    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:25.984349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.984188    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:25.984349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:25.984199    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0e46fada-1bd0-4b22-96ab-3c49c4b3199e-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:26.237520 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:26.237487    4456 generic.go:356] "Generic (PLEG): container finished" podID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" containerID="b8e26c4e4edaaffcb5446aa0eb600b76142fc6e40a41d43c05253afd1e946b94" exitCode=0
Nov 02 17:55:26.237792 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:26.237551    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"
Nov 02 17:55:26.237792 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:26.237576    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6" event={"ID":"0e46fada-1bd0-4b22-96ab-3c49c4b3199e","Type":"ContainerDied","Data":"b8e26c4e4edaaffcb5446aa0eb600b76142fc6e40a41d43c05253afd1e946b94"}
Nov 02 17:55:26.237792 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:26.237635    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6" event={"ID":"0e46fada-1bd0-4b22-96ab-3c49c4b3199e","Type":"ContainerDied","Data":"82d5c328a8a272ae602806e4c4454bc5b032f44d31e955b664f22927e524a38b"}
Nov 02 17:55:26.237792 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:26.237662    4456 scope.go:122] "RemoveContainer" containerID="b8e26c4e4edaaffcb5446aa0eb600b76142fc6e40a41d43c05253afd1e946b94"
Nov 02 17:55:26.269023 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:26.269000    4456 scope.go:122] "RemoveContainer" containerID="131e4bce0b28eec82743c5f5c4e6857572a311fdf264f6220cebc081de88ca3e"
Nov 02 17:55:26.280298 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:26.280276    4456 scope.go:122] "RemoveContainer" containerID="b8e26c4e4edaaffcb5446aa0eb600b76142fc6e40a41d43c05253afd1e946b94"
Nov 02 17:55:26.280618 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:55:26.280590    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b8e26c4e4edaaffcb5446aa0eb600b76142fc6e40a41d43c05253afd1e946b94\": container with ID starting with b8e26c4e4edaaffcb5446aa0eb600b76142fc6e40a41d43c05253afd1e946b94 not found: ID does not exist" containerID="b8e26c4e4edaaffcb5446aa0eb600b76142fc6e40a41d43c05253afd1e946b94"
Nov 02 17:55:26.280717 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:26.280632    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b8e26c4e4edaaffcb5446aa0eb600b76142fc6e40a41d43c05253afd1e946b94"} err="failed to get container status \"b8e26c4e4edaaffcb5446aa0eb600b76142fc6e40a41d43c05253afd1e946b94\": rpc error: code = NotFound desc = could not find container \"b8e26c4e4edaaffcb5446aa0eb600b76142fc6e40a41d43c05253afd1e946b94\": container with ID starting with b8e26c4e4edaaffcb5446aa0eb600b76142fc6e40a41d43c05253afd1e946b94 not found: ID does not exist"
Nov 02 17:55:26.280717 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:26.280657    4456 scope.go:122] "RemoveContainer" containerID="131e4bce0b28eec82743c5f5c4e6857572a311fdf264f6220cebc081de88ca3e"
Nov 02 17:55:26.281049 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:55:26.281023    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"131e4bce0b28eec82743c5f5c4e6857572a311fdf264f6220cebc081de88ca3e\": container with ID starting with 131e4bce0b28eec82743c5f5c4e6857572a311fdf264f6220cebc081de88ca3e not found: ID does not exist" containerID="131e4bce0b28eec82743c5f5c4e6857572a311fdf264f6220cebc081de88ca3e"
Nov 02 17:55:26.281188 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:26.281061    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"131e4bce0b28eec82743c5f5c4e6857572a311fdf264f6220cebc081de88ca3e"} err="failed to get container status \"131e4bce0b28eec82743c5f5c4e6857572a311fdf264f6220cebc081de88ca3e\": rpc error: code = NotFound desc = could not find container \"131e4bce0b28eec82743c5f5c4e6857572a311fdf264f6220cebc081de88ca3e\": container with ID starting with 131e4bce0b28eec82743c5f5c4e6857572a311fdf264f6220cebc081de88ca3e not found: ID does not exist"
Nov 02 17:55:26.308635 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:26.308606    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"]
Nov 02 17:55:26.312878 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:26.312852    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-mt-04e682fe-openshift-default-678cdc97c7-fk6g6"]
Nov 02 17:55:27.248471 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:27.248422    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:27.616998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:27.616919    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" path="/var/lib/kubelet/pods/0e46fada-1bd0-4b22-96ab-3c49c4b3199e/volumes"
Nov 02 17:55:29.139702 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.139670    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:29.296773 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.296688    4456 generic.go:356] "Generic (PLEG): container finished" podID="e37acee8-ead8-40d3-8d09-160e87271744" containerID="edc469400bd64078521d131783582abbfa00e459114777a8e5f0ea23e411cb08" exitCode=0
Nov 02 17:55:29.297152 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.296764    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq" event={"ID":"e37acee8-ead8-40d3-8d09-160e87271744","Type":"ContainerDied","Data":"edc469400bd64078521d131783582abbfa00e459114777a8e5f0ea23e411cb08"}
Nov 02 17:55:29.297265 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.297204    4456 scope.go:122] "RemoveContainer" containerID="9b2e7216425c06bb756659c307e4b41ecc390921f7e7809ec975eebf56da772c"
Nov 02 17:55:29.359506 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.359478    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:29.396326 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.396295    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:55:29.509512 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.509474    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:29.572689 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.572611    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e37acee8-ead8-40d3-8d09-160e87271744-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37acee8-ead8-40d3-8d09-160e87271744-istiod-ca-cert\") pod \"e37acee8-ead8-40d3-8d09-160e87271744\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") "
Nov 02 17:55:29.572689 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.572653    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-workload-socket\") pod \"e37acee8-ead8-40d3-8d09-160e87271744\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") "
Nov 02 17:55:29.572897 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.572783    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-workload-certs\") pod \"e37acee8-ead8-40d3-8d09-160e87271744\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") "
Nov 02 17:55:29.572936 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.572921    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-istio-envoy\") pod \"e37acee8-ead8-40d3-8d09-160e87271744\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") "
Nov 02 17:55:29.572970 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.572944    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-credential-socket\") pod \"e37acee8-ead8-40d3-8d09-160e87271744\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") "
Nov 02 17:55:29.573007 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.572980    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37acee8-ead8-40d3-8d09-160e87271744-istio-token\" (UniqueName: \"kubernetes.io/projected/e37acee8-ead8-40d3-8d09-160e87271744-istio-token\") pod \"e37acee8-ead8-40d3-8d09-160e87271744\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") "
Nov 02 17:55:29.573075 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.573031    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-workload-socket" pod "e37acee8-ead8-40d3-8d09-160e87271744" (UID: "e37acee8-ead8-40d3-8d09-160e87271744"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:29.573075 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.573065    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e37acee8-ead8-40d3-8d09-160e87271744-kube-api-access-46l26\" (UniqueName: \"kubernetes.io/projected/e37acee8-ead8-40d3-8d09-160e87271744-kube-api-access-46l26\") pod \"e37acee8-ead8-40d3-8d09-160e87271744\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") "
Nov 02 17:55:29.573352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.573077    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-workload-certs" pod "e37acee8-ead8-40d3-8d09-160e87271744" (UID: "e37acee8-ead8-40d3-8d09-160e87271744"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:29.573352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.573112    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e37acee8-ead8-40d3-8d09-160e87271744-istiod-ca-cert" pod "e37acee8-ead8-40d3-8d09-160e87271744" (UID: "e37acee8-ead8-40d3-8d09-160e87271744"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:29.573352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.573200    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-istio-data\") pod \"e37acee8-ead8-40d3-8d09-160e87271744\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") "
Nov 02 17:55:29.573352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.573254    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-credential-socket" pod "e37acee8-ead8-40d3-8d09-160e87271744" (UID: "e37acee8-ead8-40d3-8d09-160e87271744"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:29.573352 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.573272    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e37acee8-ead8-40d3-8d09-160e87271744-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37acee8-ead8-40d3-8d09-160e87271744-istio-podinfo\") pod \"e37acee8-ead8-40d3-8d09-160e87271744\" (UID: \"e37acee8-ead8-40d3-8d09-160e87271744\") "
Nov 02 17:55:29.573903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.573419    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-istio-data" pod "e37acee8-ead8-40d3-8d09-160e87271744" (UID: "e37acee8-ead8-40d3-8d09-160e87271744"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:29.574263 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.574238    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e37acee8-ead8-40d3-8d09-160e87271744-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:29.574364 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.574268    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:29.574364 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.574285    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:29.574364 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.574306    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:29.574364 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.574320    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:29.575299 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.575269    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-istio-envoy" pod "e37acee8-ead8-40d3-8d09-160e87271744" (UID: "e37acee8-ead8-40d3-8d09-160e87271744"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:29.575451 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.575436    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37acee8-ead8-40d3-8d09-160e87271744-istio-token" pod "e37acee8-ead8-40d3-8d09-160e87271744" (UID: "e37acee8-ead8-40d3-8d09-160e87271744"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:29.575521 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.575469    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e37acee8-ead8-40d3-8d09-160e87271744-istio-podinfo" pod "e37acee8-ead8-40d3-8d09-160e87271744" (UID: "e37acee8-ead8-40d3-8d09-160e87271744"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:55:29.575614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.575593    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e37acee8-ead8-40d3-8d09-160e87271744-kube-api-access-46l26" pod "e37acee8-ead8-40d3-8d09-160e87271744" (UID: "e37acee8-ead8-40d3-8d09-160e87271744"). InnerVolumeSpecName "kube-api-access-46l26". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:29.676410 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.676377    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e37acee8-ead8-40d3-8d09-160e87271744-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:29.676410 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.676409    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-46l26\" (UniqueName: \"kubernetes.io/projected/e37acee8-ead8-40d3-8d09-160e87271744-kube-api-access-46l26\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:29.676410 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.676420    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e37acee8-ead8-40d3-8d09-160e87271744-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:29.676640 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:29.676428    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e37acee8-ead8-40d3-8d09-160e87271744-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:30.310034 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:30.310001    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"
Nov 02 17:55:30.310492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:30.309996    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq" event={"ID":"e37acee8-ead8-40d3-8d09-160e87271744","Type":"ContainerDied","Data":"4fecc905f5856a6bcd63ec919ff4fd815425823acf0332dabf3a7765f005e8e8"}
Nov 02 17:55:30.310492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:30.310149    4456 scope.go:122] "RemoveContainer" containerID="edc469400bd64078521d131783582abbfa00e459114777a8e5f0ea23e411cb08"
Nov 02 17:55:30.414030 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:30.413999    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"]
Nov 02 17:55:30.438076 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:30.438039    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-1e5fcb-openshift-default-7f98f76db7-clhbq"]
Nov 02 17:55:31.210877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.210828    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:31.242330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.242292    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"]
Nov 02 17:55:31.243114 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.243097    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e37acee8-ead8-40d3-8d09-160e87271744" containerName="istio-proxy"
Nov 02 17:55:31.243192 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.243116    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e37acee8-ead8-40d3-8d09-160e87271744" containerName="istio-proxy"
Nov 02 17:55:31.243192 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.243136    4456 cpu_manager.go:414] "containerMap: removing container" podUID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" containerName="istio-proxy"
Nov 02 17:55:31.243192 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.243141    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" containerName="istio-proxy"
Nov 02 17:55:31.243192 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.243174    4456 cpu_manager.go:414] "containerMap: removing container" podUID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" containerName="istio-proxy"
Nov 02 17:55:31.243192 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.243179    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" containerName="istio-proxy"
Nov 02 17:55:31.243392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.243284    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e37acee8-ead8-40d3-8d09-160e87271744" containerName="istio-proxy"
Nov 02 17:55:31.243392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.243291    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" containerName="istio-proxy"
Nov 02 17:55:31.243392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.243302    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="0e46fada-1bd0-4b22-96ab-3c49c4b3199e" containerName="istio-proxy"
Nov 02 17:55:31.243392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.243312    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e37acee8-ead8-40d3-8d09-160e87271744" containerName="istio-proxy"
Nov 02 17:55:31.247148 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.247129    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"
Nov 02 17:55:31.265393 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.265362    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"]
Nov 02 17:55:31.272485 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.272462    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bb6d2e\""
Nov 02 17:55:31.311285 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.311247    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr\" (UID: \"f4ee5c50-5738-496a-b5ef-c86ead76e0f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"
Nov 02 17:55:31.311793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.311363    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-crv8x\" (UniqueName: \"kubernetes.io/projected/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-kube-api-access-crv8x\") pod \"payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr\" (UID: \"f4ee5c50-5738-496a-b5ef-c86ead76e0f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"
Nov 02 17:55:31.429867 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.428155    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr\" (UID: \"f4ee5c50-5738-496a-b5ef-c86ead76e0f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"
Nov 02 17:55:31.429867 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.428266    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-crv8x\" (UniqueName: \"kubernetes.io/projected/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-kube-api-access-crv8x\") pod \"payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr\" (UID: \"f4ee5c50-5738-496a-b5ef-c86ead76e0f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"
Nov 02 17:55:31.429867 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.429708    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr\" (UID: \"f4ee5c50-5738-496a-b5ef-c86ead76e0f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"
Nov 02 17:55:31.444028 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.443807    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:31.510625 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.510505    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-crv8x\" (UniqueName: \"kubernetes.io/projected/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-kube-api-access-crv8x\") pod \"payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr\" (UID: \"f4ee5c50-5738-496a-b5ef-c86ead76e0f6\") " pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"
Nov 02 17:55:31.563379 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.563087    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"
Nov 02 17:55:31.732996 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:31.732960    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e37acee8-ead8-40d3-8d09-160e87271744" path="/var/lib/kubelet/pods/e37acee8-ead8-40d3-8d09-160e87271744/volumes"
Nov 02 17:55:32.161992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.160870    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"]
Nov 02 17:55:32.187905 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.162163    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e37acee8-ead8-40d3-8d09-160e87271744" containerName="istio-proxy"
Nov 02 17:55:32.187905 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.162190    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e37acee8-ead8-40d3-8d09-160e87271744" containerName="istio-proxy"
Nov 02 17:55:32.187905 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.169072    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.216293 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.215384    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a29999-openshift-default-dockercfg-6b7bh\""
Nov 02 17:55:32.217253 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.217226    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"]
Nov 02 17:55:32.218170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.218145    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"]
Nov 02 17:55:32.243724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.243229    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:32.274145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.270310    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ba53b4-e91a-4299-9498-f5abcd34576b-istiod-ca-cert\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.274145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.270358    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-podinfo\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.274145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.270717    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-envoy\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.274145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.270768    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-credential-socket\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.274145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.270800    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-workload-socket\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.274145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.270854    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-workload-certs\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.274145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.270895    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-data\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.274145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.270985    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gv7wn\" (UniqueName: \"kubernetes.io/projected/82ba53b4-e91a-4299-9498-f5abcd34576b-kube-api-access-gv7wn\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.274145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.271044    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-token\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.368241    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" event={"ID":"f4ee5c50-5738-496a-b5ef-c86ead76e0f6","Type":"ContainerStarted","Data":"877445f9d12eef4c52383326ec19ff97cf66c0b2f107c3c3736b9e41fcd66057"}
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.373673    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-envoy\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.373746    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-credential-socket\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.373794    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-workload-socket\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.373861    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-workload-certs\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.373922    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-data\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.374038    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gv7wn\" (UniqueName: \"kubernetes.io/projected/82ba53b4-e91a-4299-9498-f5abcd34576b-kube-api-access-gv7wn\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.374116    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-token\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.374278    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ba53b4-e91a-4299-9498-f5abcd34576b-istiod-ca-cert\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.374332    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-podinfo\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.375422    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-workload-socket\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.376071    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-data\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.377030    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ba53b4-e91a-4299-9498-f5abcd34576b-istiod-ca-cert\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.377739    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-credential-socket\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.378428    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-workload-certs\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.383222    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-podinfo\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.387392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.387307    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-envoy\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.403231 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.403196    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-token\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.421909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.420221    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gv7wn\" (UniqueName: \"kubernetes.io/projected/82ba53b4-e91a-4299-9498-f5abcd34576b-kube-api-access-gv7wn\") pod \"e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") " pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.586907 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.586140    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:32.616898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.613735    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:32.970278 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:55:32.970234    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod82ba53b4_e91a_4299_9498_f5abcd34576b.slice/crio-34501ef515064b29330a7fee7503fbc73cfb72b427115992b8ff7db7323b0c24 WatchSource:0}: Error finding container 34501ef515064b29330a7fee7503fbc73cfb72b427115992b8ff7db7323b0c24: Status 404 returned error can't find the container with id 34501ef515064b29330a7fee7503fbc73cfb72b427115992b8ff7db7323b0c24
Nov 02 17:55:32.970552 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.970528    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"]
Nov 02 17:55:32.974433 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.974393    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:32.974553 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.974478    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:32.974553 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:32.974521    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:33.392562 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:33.392451    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" event={"ID":"f4ee5c50-5738-496a-b5ef-c86ead76e0f6","Type":"ContainerStarted","Data":"52131bd7b83ca9822e72e558a3dff218d4a6e09dcbe0ddc00be3cd9e069c1cbb"}
Nov 02 17:55:33.393227 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:33.393198    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"
Nov 02 17:55:33.394537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:33.394509    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" event={"ID":"82ba53b4-e91a-4299-9498-f5abcd34576b","Type":"ContainerStarted","Data":"0b5a297c1cf1ce7307713189a5bb582c30e4186d062817fc68e564f7bf4411ba"}
Nov 02 17:55:33.394635 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:33.394559    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" event={"ID":"82ba53b4-e91a-4299-9498-f5abcd34576b","Type":"ContainerStarted","Data":"34501ef515064b29330a7fee7503fbc73cfb72b427115992b8ff7db7323b0c24"}
Nov 02 17:55:33.487160 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:33.487123    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:33.491656 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:33.491601    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" podStartSLOduration=2.491583421 podStartE2EDuration="2.491583421s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:33.442011555 +0000 UTC m=+1549.933074715" watchObservedRunningTime="2026-11-02 17:55:33.491583421 +0000 UTC m=+1549.982646583"
Nov 02 17:55:33.495515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:33.494346    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" podStartSLOduration=1.494315753 podStartE2EDuration="1.494315753s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:33.491471641 +0000 UTC m=+1549.982534807" watchObservedRunningTime="2026-11-02 17:55:33.494315753 +0000 UTC m=+1549.985378915"
Nov 02 17:55:33.587502 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:33.587467    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:34.220450 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:34.220397    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"
Nov 02 17:55:34.389401 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:34.389356    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:34.573587 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:34.572116    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:34.595832 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:34.595783    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:35.420198 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:35.420166    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:35.421126 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:35.421103    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:55:35.460614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:35.460580    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:35.722667 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:35.722633    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:36.054311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:36.054192    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:36.255438 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:36.255405    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:37.632238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:37.632193    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:37.798919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:37.798876    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:38.632394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:38.632357    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:38.786323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:38.786279    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"]
Nov 02 17:55:38.790554 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:38.790533    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"
Nov 02 17:55:38.809422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:38.809390    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:38.815365 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:38.815342    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a29999\""
Nov 02 17:55:38.818611 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:38.818582    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"]
Nov 02 17:55:38.882098 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:38.878678    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a29999-dockercfg-r7mt7\""
Nov 02 17:55:38.913469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:38.909976    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"]
Nov 02 17:55:39.009835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:38.997462    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 17:55:39.009835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.003223    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f586g\" (UniqueName: \"kubernetes.io/projected/7e09749e-e648-40a3-a5f1-f0dc35fccf24-kube-api-access-f586g\") pod \"payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45\" (UID: \"7e09749e-e648-40a3-a5f1-f0dc35fccf24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"
Nov 02 17:55:39.009835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.003373    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45\" (UID: \"7e09749e-e648-40a3-a5f1-f0dc35fccf24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"
Nov 02 17:55:39.115530 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.115214    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f586g\" (UniqueName: \"kubernetes.io/projected/7e09749e-e648-40a3-a5f1-f0dc35fccf24-kube-api-access-f586g\") pod \"payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45\" (UID: \"7e09749e-e648-40a3-a5f1-f0dc35fccf24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"
Nov 02 17:55:39.115530 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.115353    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45\" (UID: \"7e09749e-e648-40a3-a5f1-f0dc35fccf24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"
Nov 02 17:55:39.117424 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.116886    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45\" (UID: \"7e09749e-e648-40a3-a5f1-f0dc35fccf24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"
Nov 02 17:55:39.137542 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.137510    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"]
Nov 02 17:55:39.142409 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.142378    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f586g\" (UniqueName: \"kubernetes.io/projected/7e09749e-e648-40a3-a5f1-f0dc35fccf24-kube-api-access-f586g\") pod \"payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45\" (UID: \"7e09749e-e648-40a3-a5f1-f0dc35fccf24\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"
Nov 02 17:55:39.217091 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.217057    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b4g6z\" (UniqueName: \"kubernetes.io/projected/1c190514-ba33-4844-a9ec-e4bb47b5f706-kube-api-access-b4g6z\") pod \"payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6\" (UID: \"1c190514-ba33-4844-a9ec-e4bb47b5f706\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 17:55:39.217507 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.217476    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6\" (UID: \"1c190514-ba33-4844-a9ec-e4bb47b5f706\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 17:55:39.319521 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.319487    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b4g6z\" (UniqueName: \"kubernetes.io/projected/1c190514-ba33-4844-a9ec-e4bb47b5f706-kube-api-access-b4g6z\") pod \"payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6\" (UID: \"1c190514-ba33-4844-a9ec-e4bb47b5f706\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 17:55:39.319864 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.319837    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6\" (UID: \"1c190514-ba33-4844-a9ec-e4bb47b5f706\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 17:55:39.320743 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.320722    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6\" (UID: \"1c190514-ba33-4844-a9ec-e4bb47b5f706\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 17:55:39.344077 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.344045    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b4g6z\" (UniqueName: \"kubernetes.io/projected/1c190514-ba33-4844-a9ec-e4bb47b5f706-kube-api-access-b4g6z\") pod \"payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6\" (UID: \"1c190514-ba33-4844-a9ec-e4bb47b5f706\") " pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 17:55:39.406866 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.406830    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"
Nov 02 17:55:39.441124 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.441096    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 17:55:39.759324 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:55:39.759292    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e09749e_e648_40a3_a5f1_f0dc35fccf24.slice/crio-27834b12d06e3c3b988ff0cd49935574491e3c10b89279f045db0b5cce79e9b9 WatchSource:0}: Error finding container 27834b12d06e3c3b988ff0cd49935574491e3c10b89279f045db0b5cce79e9b9: Status 404 returned error can't find the container with id 27834b12d06e3c3b988ff0cd49935574491e3c10b89279f045db0b5cce79e9b9
Nov 02 17:55:39.759758 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.759674    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"]
Nov 02 17:55:39.809001 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:55:39.808964    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1c190514_ba33_4844_a9ec_e4bb47b5f706.slice/crio-e2874138d3aa9323123ddec8ad7186f666235e1a9186836d7e0aed34dc66cb17 WatchSource:0}: Error finding container e2874138d3aa9323123ddec8ad7186f666235e1a9186836d7e0aed34dc66cb17: Status 404 returned error can't find the container with id e2874138d3aa9323123ddec8ad7186f666235e1a9186836d7e0aed34dc66cb17
Nov 02 17:55:39.811656 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.811626    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"]
Nov 02 17:55:39.849795 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:39.849747    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:40.478779 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:40.478735    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" event={"ID":"7e09749e-e648-40a3-a5f1-f0dc35fccf24","Type":"ContainerStarted","Data":"8c6acffecf9ac35beb62fec780633f1baf430b68dfeefab50a43468a043033a3"}
Nov 02 17:55:40.479008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:40.478885    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" event={"ID":"7e09749e-e648-40a3-a5f1-f0dc35fccf24","Type":"ContainerStarted","Data":"27834b12d06e3c3b988ff0cd49935574491e3c10b89279f045db0b5cce79e9b9"}
Nov 02 17:55:40.479532 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:40.479505    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"
Nov 02 17:55:40.484857 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:40.484800    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" event={"ID":"1c190514-ba33-4844-a9ec-e4bb47b5f706","Type":"ContainerStarted","Data":"e8fac6b06137604d986996a69eb0437546b5e8e988303a1180286636239ba239"}
Nov 02 17:55:40.485017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:40.484882    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" event={"ID":"1c190514-ba33-4844-a9ec-e4bb47b5f706","Type":"ContainerStarted","Data":"e2874138d3aa9323123ddec8ad7186f666235e1a9186836d7e0aed34dc66cb17"}
Nov 02 17:55:40.485157 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:40.485137    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 17:55:40.523581 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:40.523513    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" podStartSLOduration=2.523494273 podStartE2EDuration="2.523494273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:40.521965197 +0000 UTC m=+1557.013028356" watchObservedRunningTime="2026-11-02 17:55:40.523494273 +0000 UTC m=+1557.014557435"
Nov 02 17:55:40.570943 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:40.570897    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" podStartSLOduration=2.5708834080000003 podStartE2EDuration="2.570883408s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:40.565951016 +0000 UTC m=+1557.057014180" watchObservedRunningTime="2026-11-02 17:55:40.570883408 +0000 UTC m=+1557.061946568"
Nov 02 17:55:41.906243 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:41.906201    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:43.838080 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:43.838046    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"]
Nov 02 17:55:43.842340 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:43.842316    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:43.856642 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:43.856619    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:43.880493 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:43.874943    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-063f48-openshift-default-dockercfg-n95mk\""
Nov 02 17:55:43.900558 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:43.886159    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"]
Nov 02 17:55:43.926479 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:43.918025    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-podinfo\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:43.926479 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:43.918094    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab077054-0cce-44bd-aecc-ee2938d106c9-istiod-ca-cert\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:43.926479 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:43.918139    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-envoy\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:43.926479 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:43.918243    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2q74n\" (UniqueName: \"kubernetes.io/projected/ab077054-0cce-44bd-aecc-ee2938d106c9-kube-api-access-2q74n\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:43.926479 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:43.918291    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-data\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:43.926479 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:43.918350    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-workload-certs\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:43.926479 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:43.918392    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-workload-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:43.926479 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:43.918415    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-credential-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:43.926479 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:43.918567    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-token\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.020442 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.020402    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-podinfo\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.020603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.020504    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab077054-0cce-44bd-aecc-ee2938d106c9-istiod-ca-cert\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.020603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.020549    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-envoy\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.020740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.020637    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2q74n\" (UniqueName: \"kubernetes.io/projected/ab077054-0cce-44bd-aecc-ee2938d106c9-kube-api-access-2q74n\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.020740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.020678    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-data\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.020996 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.020972    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-workload-certs\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.021205 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.021185    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-workload-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.021324 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.021238    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-workload-certs\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.021481 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.021365    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab077054-0cce-44bd-aecc-ee2938d106c9-istiod-ca-cert\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.021623 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.021533    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-workload-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.021722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.021314    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-credential-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.021722 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.021721    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-data\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.021971 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.021691    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-credential-socket\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.022055 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.021973    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-token\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.023338 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.023315    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-envoy\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.023631 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.023612    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-podinfo\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.045215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.045187    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2q74n\" (UniqueName: \"kubernetes.io/projected/ab077054-0cce-44bd-aecc-ee2938d106c9-kube-api-access-2q74n\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.045710 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.045692    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-token\") pod \"e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") " pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.158632 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.158539    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:44.415234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.415138    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"
Nov 02 17:55:44.519159 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.519128    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"]
Nov 02 17:55:44.522834 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:55:44.522792    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podab077054_0cce_44bd_aecc_ee2938d106c9.slice/crio-4815dcdafe4f45341ef465a8e4f493cb952eec85b40209ed040c51655f16503b WatchSource:0}: Error finding container 4815dcdafe4f45341ef465a8e4f493cb952eec85b40209ed040c51655f16503b: Status 404 returned error can't find the container with id 4815dcdafe4f45341ef465a8e4f493cb952eec85b40209ed040c51655f16503b
Nov 02 17:55:44.527040 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.527001    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:44.527558 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.527532    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:44.527715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:44.527697    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:55:45.547854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:45.547783    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l" event={"ID":"ab077054-0cce-44bd-aecc-ee2938d106c9","Type":"ContainerStarted","Data":"993c47b2e4ab9a8bec71be49a1d715eb16bb30e2cfae42e995f4ea0dc51b11d2"}
Nov 02 17:55:45.548418 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:45.547895    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l" event={"ID":"ab077054-0cce-44bd-aecc-ee2938d106c9","Type":"ContainerStarted","Data":"4815dcdafe4f45341ef465a8e4f493cb952eec85b40209ed040c51655f16503b"}
Nov 02 17:55:45.621182 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:45.621116    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l" podStartSLOduration=2.621097617 podStartE2EDuration="2.621097617s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:55:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:55:45.609149789 +0000 UTC m=+1562.100213003" watchObservedRunningTime="2026-11-02 17:55:45.621097617 +0000 UTC m=+1562.112160782"
Nov 02 17:55:45.644208 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:45.644167    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:46.159119 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:46.159087    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:46.161042 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:46.161016    4456 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 17:55:46.161134 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:46.161073    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l" podUID="ab077054-0cce-44bd-aecc-ee2938d106c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 17:55:46.167113 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:55:46.167078    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:55:46.762456 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:46.762417    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:46.919329 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:46.919276    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:47.159469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:47.159391    4456 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 17:55:47.159469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:47.159450    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l" podUID="ab077054-0cce-44bd-aecc-ee2938d106c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 17:55:47.421299 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:47.420285    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:48.159651 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:48.159617    4456 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 17:55:48.160118 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:48.159669    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l" podUID="ab077054-0cce-44bd-aecc-ee2938d106c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 17:55:48.477960 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:48.477927    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"]
Nov 02 17:55:48.478150 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:48.478124    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s" podUID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" containerName="istio-proxy" containerID="cri-o://b6c29b54d2e8809522ee97f272b9dfd8c889bc72413ca91c1099b08a1dd31041" gracePeriod=30
Nov 02 17:55:48.531903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:48.528714    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"]
Nov 02 17:55:48.531903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:48.528971    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy" containerID="cri-o://b93fc8271576e0f58e400c54457f3aa6226f1bed6d4e87a7f2447ca8db7e58fe" gracePeriod=30
Nov 02 17:55:49.162053 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:49.161607    4456 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 17:55:49.162053 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:49.161666    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l" podUID="ab077054-0cce-44bd-aecc-ee2938d106c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 17:55:49.275835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:49.274654    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:50.159232 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:50.159181    4456 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused" start-of-body=
Nov 02 17:55:50.159408 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:50.159249    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l" podUID="ab077054-0cce-44bd-aecc-ee2938d106c9" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.177:15021/healthz/ready\": dial tcp 10.134.0.177:15021: connect: connection refused"
Nov 02 17:55:50.953653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:50.953620    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:51.256622 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:51.256528    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:51.257141 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:51.257102    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:51.258477 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:51.258458    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 17:55:51.506988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:51.506908    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 17:55:51.511950 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:51.511925    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"
Nov 02 17:55:52.804089 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:52.804028    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:53.354703 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:53.354664    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:55:53.354892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:53.354724    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s" podUID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:55:53.694870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:53.694428    4456 generic.go:356] "Generic (PLEG): container finished" podID="54fe85b3-015f-4145-abba-9d31009c6229" containerID="b93fc8271576e0f58e400c54457f3aa6226f1bed6d4e87a7f2447ca8db7e58fe" exitCode=0
Nov 02 17:55:53.694870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:53.694629    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" event={"ID":"54fe85b3-015f-4145-abba-9d31009c6229","Type":"ContainerDied","Data":"b93fc8271576e0f58e400c54457f3aa6226f1bed6d4e87a7f2447ca8db7e58fe"}
Nov 02 17:55:53.711017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:53.710972    4456 generic.go:356] "Generic (PLEG): container finished" podID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" containerID="b6c29b54d2e8809522ee97f272b9dfd8c889bc72413ca91c1099b08a1dd31041" exitCode=0
Nov 02 17:55:53.711383 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:53.711351    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s" event={"ID":"06c8d0c2-d05a-4ece-a583-bfa1aed218ce","Type":"ContainerDied","Data":"b6c29b54d2e8809522ee97f272b9dfd8c889bc72413ca91c1099b08a1dd31041"}
Nov 02 17:55:53.711594 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:53.711578    4456 scope.go:122] "RemoveContainer" containerID="342f9a8ed8df1f4b2c22ebbefea71c3f8a51cdb1349cb8424e3f22803605ac5c"
Nov 02 17:55:53.886514 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:53.886487    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:55:53.932406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:53.932380    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:55:54.002367 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.000621    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-podinfo\") pod \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") "
Nov 02 17:55:54.002367 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.000721    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-workload-socket\") pod \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") "
Nov 02 17:55:54.002367 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.000770    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-envoy\") pod \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") "
Nov 02 17:55:54.002367 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.000803    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-token\" (UniqueName: \"kubernetes.io/projected/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-token\") pod \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") "
Nov 02 17:55:54.002367 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.000983    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-data\") pod \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") "
Nov 02 17:55:54.002367 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.001060    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istiod-ca-cert\") pod \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") "
Nov 02 17:55:54.002367 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.001111    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-credential-socket\") pod \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") "
Nov 02 17:55:54.002367 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.001262    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-workload-certs\") pod \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") "
Nov 02 17:55:54.002367 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.001352    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-kube-api-access-j79jc\" (UniqueName: \"kubernetes.io/projected/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-kube-api-access-j79jc\") pod \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\" (UID: \"06c8d0c2-d05a-4ece-a583-bfa1aed218ce\") "
Nov 02 17:55:54.005251 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.005161    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-kube-api-access-j79jc" pod "06c8d0c2-d05a-4ece-a583-bfa1aed218ce" (UID: "06c8d0c2-d05a-4ece-a583-bfa1aed218ce"). InnerVolumeSpecName "kube-api-access-j79jc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:54.005396 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.005326    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j79jc\" (UniqueName: \"kubernetes.io/projected/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-kube-api-access-j79jc\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.005582 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.005559    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-data" pod "06c8d0c2-d05a-4ece-a583-bfa1aed218ce" (UID: "06c8d0c2-d05a-4ece-a583-bfa1aed218ce"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:54.005783 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.005762    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-workload-certs" pod "06c8d0c2-d05a-4ece-a583-bfa1aed218ce" (UID: "06c8d0c2-d05a-4ece-a583-bfa1aed218ce"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:54.006041 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.006014    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-workload-socket" pod "06c8d0c2-d05a-4ece-a583-bfa1aed218ce" (UID: "06c8d0c2-d05a-4ece-a583-bfa1aed218ce"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:54.006131 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.006102    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istiod-ca-cert" pod "06c8d0c2-d05a-4ece-a583-bfa1aed218ce" (UID: "06c8d0c2-d05a-4ece-a583-bfa1aed218ce"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:54.006315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.006277    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-credential-socket" pod "06c8d0c2-d05a-4ece-a583-bfa1aed218ce" (UID: "06c8d0c2-d05a-4ece-a583-bfa1aed218ce"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:54.008725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.008694    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-podinfo" pod "06c8d0c2-d05a-4ece-a583-bfa1aed218ce" (UID: "06c8d0c2-d05a-4ece-a583-bfa1aed218ce"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:55:54.013959 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.013932    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-envoy" pod "06c8d0c2-d05a-4ece-a583-bfa1aed218ce" (UID: "06c8d0c2-d05a-4ece-a583-bfa1aed218ce"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:54.015576 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.015542    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-token" pod "06c8d0c2-d05a-4ece-a583-bfa1aed218ce" (UID: "06c8d0c2-d05a-4ece-a583-bfa1aed218ce"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:54.105952 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.105914    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/54fe85b3-015f-4145-abba-9d31009c6229-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fe85b3-015f-4145-abba-9d31009c6229-istio-podinfo\") pod \"54fe85b3-015f-4145-abba-9d31009c6229\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") "
Nov 02 17:55:54.106150 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.106024    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-workload-socket\") pod \"54fe85b3-015f-4145-abba-9d31009c6229\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") "
Nov 02 17:55:54.106150 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.106101    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fe85b3-015f-4145-abba-9d31009c6229-istio-token\" (UniqueName: \"kubernetes.io/projected/54fe85b3-015f-4145-abba-9d31009c6229-istio-token\") pod \"54fe85b3-015f-4145-abba-9d31009c6229\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") "
Nov 02 17:55:54.106273 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.106260    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-istio-envoy\") pod \"54fe85b3-015f-4145-abba-9d31009c6229\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") "
Nov 02 17:55:54.106653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.106333    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/54fe85b3-015f-4145-abba-9d31009c6229-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fe85b3-015f-4145-abba-9d31009c6229-istiod-ca-cert\") pod \"54fe85b3-015f-4145-abba-9d31009c6229\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") "
Nov 02 17:55:54.106653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.106460    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-credential-socket\") pod \"54fe85b3-015f-4145-abba-9d31009c6229\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") "
Nov 02 17:55:54.106653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.106592    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/54fe85b3-015f-4145-abba-9d31009c6229-kube-api-access-64dlm\" (UniqueName: \"kubernetes.io/projected/54fe85b3-015f-4145-abba-9d31009c6229-kube-api-access-64dlm\") pod \"54fe85b3-015f-4145-abba-9d31009c6229\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") "
Nov 02 17:55:54.106653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.106650    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-istio-data\") pod \"54fe85b3-015f-4145-abba-9d31009c6229\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") "
Nov 02 17:55:54.106945 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.106731    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-workload-certs\") pod \"54fe85b3-015f-4145-abba-9d31009c6229\" (UID: \"54fe85b3-015f-4145-abba-9d31009c6229\") "
Nov 02 17:55:54.107617 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.107590    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.107724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.107620    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.107724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.107633    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.107724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.107646    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.107724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.107659    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.107724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.107671    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.107724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.107683    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.107724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.107703    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/06c8d0c2-d05a-4ece-a583-bfa1aed218ce-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.108187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.107930    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-workload-certs" pod "54fe85b3-015f-4145-abba-9d31009c6229" (UID: "54fe85b3-015f-4145-abba-9d31009c6229"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:54.108246 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.108206    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/54fe85b3-015f-4145-abba-9d31009c6229-istiod-ca-cert" pod "54fe85b3-015f-4145-abba-9d31009c6229" (UID: "54fe85b3-015f-4145-abba-9d31009c6229"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:55:54.108457 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.108426    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-credential-socket" pod "54fe85b3-015f-4145-abba-9d31009c6229" (UID: "54fe85b3-015f-4145-abba-9d31009c6229"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:54.108896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.108872    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-istio-data" pod "54fe85b3-015f-4145-abba-9d31009c6229" (UID: "54fe85b3-015f-4145-abba-9d31009c6229"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:54.109081 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.109056    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-workload-socket" pod "54fe85b3-015f-4145-abba-9d31009c6229" (UID: "54fe85b3-015f-4145-abba-9d31009c6229"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:54.109717 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.109656    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fe85b3-015f-4145-abba-9d31009c6229-istio-token" pod "54fe85b3-015f-4145-abba-9d31009c6229" (UID: "54fe85b3-015f-4145-abba-9d31009c6229"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:54.109717 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.109681    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-istio-envoy" pod "54fe85b3-015f-4145-abba-9d31009c6229" (UID: "54fe85b3-015f-4145-abba-9d31009c6229"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:55:54.110047 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.110017    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/54fe85b3-015f-4145-abba-9d31009c6229-istio-podinfo" pod "54fe85b3-015f-4145-abba-9d31009c6229" (UID: "54fe85b3-015f-4145-abba-9d31009c6229"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:55:54.110959 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.110925    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/54fe85b3-015f-4145-abba-9d31009c6229-kube-api-access-64dlm" pod "54fe85b3-015f-4145-abba-9d31009c6229" (UID: "54fe85b3-015f-4145-abba-9d31009c6229"). InnerVolumeSpecName "kube-api-access-64dlm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:55:54.209275 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.209240    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.209275 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.209270    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/54fe85b3-015f-4145-abba-9d31009c6229-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.209275 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.209279    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.209499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.209286    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/54fe85b3-015f-4145-abba-9d31009c6229-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.209499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.209294    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.209499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.209303    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64dlm\" (UniqueName: \"kubernetes.io/projected/54fe85b3-015f-4145-abba-9d31009c6229-kube-api-access-64dlm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.209499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.209311    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.209499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.209320    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/54fe85b3-015f-4145-abba-9d31009c6229-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.209499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.209327    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/54fe85b3-015f-4145-abba-9d31009c6229-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:55:54.720954 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.720928    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"
Nov 02 17:55:54.721172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.720955    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5" event={"ID":"54fe85b3-015f-4145-abba-9d31009c6229","Type":"ContainerDied","Data":"b9e36cb2167c1643fbb7a7a55bc5acc2dd22994a047c1b91138490b2b36186be"}
Nov 02 17:55:54.721172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.721037    4456 scope.go:122] "RemoveContainer" containerID="b93fc8271576e0f58e400c54457f3aa6226f1bed6d4e87a7f2447ca8db7e58fe"
Nov 02 17:55:54.722768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.722744    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"
Nov 02 17:55:54.722957 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.722775    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s" event={"ID":"06c8d0c2-d05a-4ece-a583-bfa1aed218ce","Type":"ContainerDied","Data":"7d47d1795bb8382ca32ce475cbee2f3d6c8dc511aa5b08da9e3750aa850b2be2"}
Nov 02 17:55:54.735615 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.735595    4456 scope.go:122] "RemoveContainer" containerID="b6c29b54d2e8809522ee97f272b9dfd8c889bc72413ca91c1099b08a1dd31041"
Nov 02 17:55:54.764408 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.764380    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:54.808907 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.805873    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"]
Nov 02 17:55:54.858034 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.858001    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-d479s"]
Nov 02 17:55:54.865304 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.865241    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"]
Nov 02 17:55:54.875840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:54.875785    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-jjnp5"]
Nov 02 17:55:55.622590 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:55.622495    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" path="/var/lib/kubelet/pods/06c8d0c2-d05a-4ece-a583-bfa1aed218ce/volumes"
Nov 02 17:55:55.623316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:55.623293    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="54fe85b3-015f-4145-abba-9d31009c6229" path="/var/lib/kubelet/pods/54fe85b3-015f-4145-abba-9d31009c6229/volumes"
Nov 02 17:55:56.570848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:56.567119    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:58.360434 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:58.360288    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:55:59.780954 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:59.778646    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"]
Nov 02 17:55:59.782139 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:59.782079    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc" podUID="8faa6946-48e8-4d72-be2c-080a0c8a83d2" containerName="maas-api" containerID="cri-o://5d7b826b2eb6942859781a8002a9eb8d2b374392a32292b13693724ceba3b1fb" gracePeriod=30
Nov 02 17:55:59.824069 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:55:59.824034    4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8faa6946_48e8_4d72_be2c_080a0c8a83d2.slice/crio-5d7b826b2eb6942859781a8002a9eb8d2b374392a32292b13693724ceba3b1fb.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 17:55:59.870251 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:55:59.870217    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:00.281792 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.281758    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:56:00.320500 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.320460    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8faa6946-48e8-4d72-be2c-080a0c8a83d2-metrics-certs\" (UniqueName: \"kubernetes.io/secret/8faa6946-48e8-4d72-be2c-080a0c8a83d2-metrics-certs\") pod \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") "
Nov 02 17:56:00.320655 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.320511    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8faa6946-48e8-4d72-be2c-080a0c8a83d2-kube-api-access-v5bss\" (UniqueName: \"kubernetes.io/projected/8faa6946-48e8-4d72-be2c-080a0c8a83d2-kube-api-access-v5bss\") pod \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") "
Nov 02 17:56:00.320713 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.320690    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8faa6946-48e8-4d72-be2c-080a0c8a83d2-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8faa6946-48e8-4d72-be2c-080a0c8a83d2-maas-api-tls\") pod \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") "
Nov 02 17:56:00.320827 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.320796    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8faa6946-48e8-4d72-be2c-080a0c8a83d2-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8faa6946-48e8-4d72-be2c-080a0c8a83d2-openshift-service-ca\") pod \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\" (UID: \"8faa6946-48e8-4d72-be2c-080a0c8a83d2\") "
Nov 02 17:56:00.321182 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.321152    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8faa6946-48e8-4d72-be2c-080a0c8a83d2-openshift-service-ca" pod "8faa6946-48e8-4d72-be2c-080a0c8a83d2" (UID: "8faa6946-48e8-4d72-be2c-080a0c8a83d2"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:56:00.321518 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.321500    4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/8faa6946-48e8-4d72-be2c-080a0c8a83d2-openshift-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:00.323557 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.323524    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8faa6946-48e8-4d72-be2c-080a0c8a83d2-maas-api-tls" pod "8faa6946-48e8-4d72-be2c-080a0c8a83d2" (UID: "8faa6946-48e8-4d72-be2c-080a0c8a83d2"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:56:00.323653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.323556    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8faa6946-48e8-4d72-be2c-080a0c8a83d2-metrics-certs" pod "8faa6946-48e8-4d72-be2c-080a0c8a83d2" (UID: "8faa6946-48e8-4d72-be2c-080a0c8a83d2"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:56:00.323653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.323624    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8faa6946-48e8-4d72-be2c-080a0c8a83d2-kube-api-access-v5bss" pod "8faa6946-48e8-4d72-be2c-080a0c8a83d2" (UID: "8faa6946-48e8-4d72-be2c-080a0c8a83d2"). InnerVolumeSpecName "kube-api-access-v5bss". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:56:00.431023 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.430652    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:00.453730 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.453313    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v5bss\" (UniqueName: \"kubernetes.io/projected/8faa6946-48e8-4d72-be2c-080a0c8a83d2-kube-api-access-v5bss\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:00.453730 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.453352    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/8faa6946-48e8-4d72-be2c-080a0c8a83d2-maas-api-tls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:00.453730 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.453367    4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/8faa6946-48e8-4d72-be2c-080a0c8a83d2-metrics-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:00.557018 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.556630    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"]
Nov 02 17:56:00.569762 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.569729    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-a-c3c112-pw52x"]
Nov 02 17:56:00.629840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.628568    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:00.774449 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.774394    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:00.782498 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.782467    4456 generic.go:356] "Generic (PLEG): container finished" podID="8faa6946-48e8-4d72-be2c-080a0c8a83d2" containerID="5d7b826b2eb6942859781a8002a9eb8d2b374392a32292b13693724ceba3b1fb" exitCode=0
Nov 02 17:56:00.783007 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.782520    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"
Nov 02 17:56:00.783007 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.782526    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc" event={"ID":"8faa6946-48e8-4d72-be2c-080a0c8a83d2","Type":"ContainerDied","Data":"5d7b826b2eb6942859781a8002a9eb8d2b374392a32292b13693724ceba3b1fb"}
Nov 02 17:56:00.783007 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.782583    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc" event={"ID":"8faa6946-48e8-4d72-be2c-080a0c8a83d2","Type":"ContainerDied","Data":"f51c962be6a561ffeb48b8790289b01dfcd0255e8d62e5384d72115831e174c2"}
Nov 02 17:56:00.783007 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.782615    4456 scope.go:122] "RemoveContainer" containerID="5d7b826b2eb6942859781a8002a9eb8d2b374392a32292b13693724ceba3b1fb"
Nov 02 17:56:00.801646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.801625    4456 scope.go:122] "RemoveContainer" containerID="5d7b826b2eb6942859781a8002a9eb8d2b374392a32292b13693724ceba3b1fb"
Nov 02 17:56:00.802921 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:56:00.802886    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d7b826b2eb6942859781a8002a9eb8d2b374392a32292b13693724ceba3b1fb\": container with ID starting with 5d7b826b2eb6942859781a8002a9eb8d2b374392a32292b13693724ceba3b1fb not found: ID does not exist" containerID="5d7b826b2eb6942859781a8002a9eb8d2b374392a32292b13693724ceba3b1fb"
Nov 02 17:56:00.803066 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.802929    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d7b826b2eb6942859781a8002a9eb8d2b374392a32292b13693724ceba3b1fb"} err="failed to get container status \"5d7b826b2eb6942859781a8002a9eb8d2b374392a32292b13693724ceba3b1fb\": rpc error: code = NotFound desc = could not find container \"5d7b826b2eb6942859781a8002a9eb8d2b374392a32292b13693724ceba3b1fb\": container with ID starting with 5d7b826b2eb6942859781a8002a9eb8d2b374392a32292b13693724ceba3b1fb not found: ID does not exist"
Nov 02 17:56:00.839416 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.839335    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"]
Nov 02 17:56:00.867126 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:00.867088    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-api-a-c3c112-7f44856d48-mgvhc"]
Nov 02 17:56:01.130553 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:01.130450    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:01.316562 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:01.316504    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:01.621358 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:01.621321    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5c11216b-95b5-42ed-bc15-fe3bc02945c7" path="/var/lib/kubelet/pods/5c11216b-95b5-42ed-bc15-fe3bc02945c7/volumes"
Nov 02 17:56:01.621919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:01.621893    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8faa6946-48e8-4d72-be2c-080a0c8a83d2" path="/var/lib/kubelet/pods/8faa6946-48e8-4d72-be2c-080a0c8a83d2/volumes"
Nov 02 17:56:02.828167 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:02.828126    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:04.542032 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:04.541993    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:06.319219 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:06.318429    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:06.533217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:06.533180    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:06.682187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:06.682103    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:08.108671 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:08.108628    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:08.568825 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:08.568780    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:08.726987 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:08.726943    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:08.907297 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:08.907201    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"]
Nov 02 17:56:08.907506 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:08.907475    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp" podUID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerName="istio-proxy" containerID="cri-o://1f3338eaa0ca5d0255f41a33c509df2ba8023c26afcd02725fd1c7f7c3299602" gracePeriod=30
Nov 02 17:56:08.920716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:08.920679    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"]
Nov 02 17:56:08.921187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:08.921157    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql" podUID="32d7a3ef-ec4e-4639-b849-baf8078f5fd0" containerName="istio-proxy" containerID="cri-o://d255c53e82a16333f04bcb424be778697aa2e2fbfb8d5e549007236f80fd5375" gracePeriod=30
Nov 02 17:56:08.976905 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:08.976874    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:09.538496 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.538459    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"]
Nov 02 17:56:09.539275 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.539257    4456 cpu_manager.go:414] "containerMap: removing container" podUID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" containerName="istio-proxy"
Nov 02 17:56:09.539327 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.539277    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" containerName="istio-proxy"
Nov 02 17:56:09.539327 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.539311    4456 cpu_manager.go:414] "containerMap: removing container" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy"
Nov 02 17:56:09.539327 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.539317    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy"
Nov 02 17:56:09.539327 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.539324    4456 cpu_manager.go:414] "containerMap: removing container" podUID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" containerName="istio-proxy"
Nov 02 17:56:09.539461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.539329    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" containerName="istio-proxy"
Nov 02 17:56:09.539461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.539337    4456 cpu_manager.go:414] "containerMap: removing container" podUID="8faa6946-48e8-4d72-be2c-080a0c8a83d2" containerName="maas-api"
Nov 02 17:56:09.539461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.539342    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8faa6946-48e8-4d72-be2c-080a0c8a83d2" containerName="maas-api"
Nov 02 17:56:09.539461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.539442    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="54fe85b3-015f-4145-abba-9d31009c6229" containerName="istio-proxy"
Nov 02 17:56:09.539461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.539454    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" containerName="istio-proxy"
Nov 02 17:56:09.539606 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.539464    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8faa6946-48e8-4d72-be2c-080a0c8a83d2" containerName="maas-api"
Nov 02 17:56:09.539606 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.539483    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="06c8d0c2-d05a-4ece-a583-bfa1aed218ce" containerName="istio-proxy"
Nov 02 17:56:09.543190 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.543174    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"
Nov 02 17:56:09.565626 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.565599    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"]
Nov 02 17:56:09.625627 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.625561    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:56:09.625921 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.625673    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:56:09.626217 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.626197    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-227833-z5rnk\" (UID: \"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"
Nov 02 17:56:09.626418 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.626393    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wqmqh\" (UniqueName: \"kubernetes.io/projected/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-kube-api-access-wqmqh\") pod \"maas-api-revoke-keys-e2e-api-b-227833-z5rnk\" (UID: \"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"
Nov 02 17:56:09.626632 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.626617    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-227833-z5rnk\" (UID: \"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"
Nov 02 17:56:09.727854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.727785    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-227833-z5rnk\" (UID: \"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"
Nov 02 17:56:09.728041 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.727872    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wqmqh\" (UniqueName: \"kubernetes.io/projected/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-kube-api-access-wqmqh\") pod \"maas-api-revoke-keys-e2e-api-b-227833-z5rnk\" (UID: \"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"
Nov 02 17:56:09.728122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.728054    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-227833-z5rnk\" (UID: \"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"
Nov 02 17:56:09.728723 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.728696    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-api-b-227833-z5rnk\" (UID: \"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"
Nov 02 17:56:09.730423 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.730404    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-api-b-227833-z5rnk\" (UID: \"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"
Nov 02 17:56:09.754751 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.754725    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wqmqh\" (UniqueName: \"kubernetes.io/projected/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-kube-api-access-wqmqh\") pod \"maas-api-revoke-keys-e2e-api-b-227833-z5rnk\" (UID: \"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"
Nov 02 17:56:09.858257 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:09.858178    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"
Nov 02 17:56:10.182795 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:10.182759    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"]
Nov 02 17:56:10.241070 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:10.241043    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:10.899641 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:10.899534    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk" event={"ID":"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3","Type":"ContainerStarted","Data":"384a689b9c5f882442305b364f7f0d010f84e512893f729c400ab5d63b7eb392"}
Nov 02 17:56:10.899641 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:10.899604    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk" event={"ID":"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3","Type":"ContainerStarted","Data":"df767410c1e9da82c64d272f6b4e669c01fe607127e93628933d8315a6ec1a80"}
Nov 02 17:56:10.948170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:10.948110    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk" podStartSLOduration=1.94809447 podStartE2EDuration="1.94809447s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:56:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:56:10.944598695 +0000 UTC m=+1587.435661856" watchObservedRunningTime="2026-11-02 17:56:10.94809447 +0000 UTC m=+1587.439157631"
Nov 02 17:56:11.910973 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:11.910931    4456 generic.go:356] "Generic (PLEG): container finished" podID="34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3" containerID="384a689b9c5f882442305b364f7f0d010f84e512893f729c400ab5d63b7eb392" exitCode=0
Nov 02 17:56:11.911416 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:11.910968    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk" event={"ID":"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3","Type":"ContainerDied","Data":"384a689b9c5f882442305b364f7f0d010f84e512893f729c400ab5d63b7eb392"}
Nov 02 17:56:12.117626 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:12.117588    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:12.272979 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:12.272941    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:13.094437 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:13.094410    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"
Nov 02 17:56:13.295787 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:13.295753    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-kube-api-access-wqmqh\" (UniqueName: \"kubernetes.io/projected/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-kube-api-access-wqmqh\") pod \"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3\" (UID: \"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3\") "
Nov 02 17:56:13.296580 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:13.296558    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-maas-api-openshift-ca\") pod \"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3\" (UID: \"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3\") "
Nov 02 17:56:13.297138 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:13.297119    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-maas-api-xks-ca\") pod \"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3\" (UID: \"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3\") "
Nov 02 17:56:13.300216 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:13.297042    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-maas-api-openshift-ca" pod "34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3" (UID: "34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:56:13.319980 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:13.319924    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-kube-api-access-wqmqh" pod "34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3" (UID: "34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3"). InnerVolumeSpecName "kube-api-access-wqmqh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:56:13.325804 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:13.325716    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-maas-api-xks-ca" pod "34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3" (UID: "34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:56:13.401415 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:13.401379    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:13.401415 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:13.401407    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wqmqh\" (UniqueName: \"kubernetes.io/projected/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-kube-api-access-wqmqh\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:13.401415 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:13.401416    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:13.749055 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:13.749014    4456 patch_prober.go:29] interesting pod/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:56:13.749238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:13.749083    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql" podUID="32d7a3ef-ec4e-4639-b849-baf8078f5fd0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:56:13.932498 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:13.932456    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk" event={"ID":"34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3","Type":"ContainerDied","Data":"df767410c1e9da82c64d272f6b4e669c01fe607127e93628933d8315a6ec1a80"}
Nov 02 17:56:13.932630 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:13.932511    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df767410c1e9da82c64d272f6b4e669c01fe607127e93628933d8315a6ec1a80"
Nov 02 17:56:13.932630 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:13.932471    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"
Nov 02 17:56:14.003315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.003280    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:14.279005 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.278975    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:56:14.323094 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.323068    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:56:14.331614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.331580    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-envoy\") pod \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") "
Nov 02 17:56:14.331786 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.331663    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istiod-ca-cert\") pod \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") "
Nov 02 17:56:14.331786 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.331697    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-podinfo\") pod \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") "
Nov 02 17:56:14.331958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.331828    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-data\") pod \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") "
Nov 02 17:56:14.331958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.331937    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-token\" (UniqueName: \"kubernetes.io/projected/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-token\") pod \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") "
Nov 02 17:56:14.332426 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.332054    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-credential-socket\") pod \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") "
Nov 02 17:56:14.332426 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.332278    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-workload-socket\") pod \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") "
Nov 02 17:56:14.332426 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.332362    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-workload-certs\") pod \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") "
Nov 02 17:56:14.332426 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.332421    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-kube-api-access-nqr47\" (UniqueName: \"kubernetes.io/projected/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-kube-api-access-nqr47\") pod \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\" (UID: \"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e\") "
Nov 02 17:56:14.332985 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.332964    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-credential-socket" pod "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" (UID: "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:14.333675 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.333651    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.333935 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.333888    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-data" pod "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" (UID: "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:14.336662 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.336637    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-token" pod "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" (UID: "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:56:14.337440 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.337415    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istiod-ca-cert" pod "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" (UID: "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:56:14.337880 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.337730    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-workload-socket" pod "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" (UID: "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:14.337989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.337954    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-workload-certs" pod "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" (UID: "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:14.338105 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.338034    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-kube-api-access-nqr47" pod "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" (UID: "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e"). InnerVolumeSpecName "kube-api-access-nqr47". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:56:14.342215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.342183    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-podinfo" pod "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" (UID: "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:56:14.347742 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.347716    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-envoy" pod "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" (UID: "a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:14.434825 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.434790    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-podinfo\") pod \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") "
Nov 02 17:56:14.435026 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.434906    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-workload-certs\") pod \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") "
Nov 02 17:56:14.435026 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.434950    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-workload-socket\") pod \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") "
Nov 02 17:56:14.435153 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.435073    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-kube-api-access-n2blw\" (UniqueName: \"kubernetes.io/projected/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-kube-api-access-n2blw\") pod \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") "
Nov 02 17:56:14.435153 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.435144    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-data\") pod \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") "
Nov 02 17:56:14.435257 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.435201    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-envoy\") pod \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") "
Nov 02 17:56:14.435308 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.435251    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-workload-certs" pod "32d7a3ef-ec4e-4639-b849-baf8078f5fd0" (UID: "32d7a3ef-ec4e-4639-b849-baf8078f5fd0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:14.435308 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.435285    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istiod-ca-cert\") pod \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") "
Nov 02 17:56:14.435370 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.435344    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-credential-socket\") pod \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") "
Nov 02 17:56:14.435444 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.435422    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-token\" (UniqueName: \"kubernetes.io/projected/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-token\") pod \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\" (UID: \"32d7a3ef-ec4e-4639-b849-baf8078f5fd0\") "
Nov 02 17:56:14.436037 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.436013    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.436037 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.436036    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqr47\" (UniqueName: \"kubernetes.io/projected/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-kube-api-access-nqr47\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.436248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.436045    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.436248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.436055    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.436248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.436063    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.436248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.436070    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.436248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.436079    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.436248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.436086    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.436248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.436093    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.436728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.436700    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-credential-socket" pod "32d7a3ef-ec4e-4639-b849-baf8078f5fd0" (UID: "32d7a3ef-ec4e-4639-b849-baf8078f5fd0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:14.436728 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.436718    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-data" pod "32d7a3ef-ec4e-4639-b849-baf8078f5fd0" (UID: "32d7a3ef-ec4e-4639-b849-baf8078f5fd0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:14.436882 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.436845    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-workload-socket" pod "32d7a3ef-ec4e-4639-b849-baf8078f5fd0" (UID: "32d7a3ef-ec4e-4639-b849-baf8078f5fd0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:14.436954 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.436904    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istiod-ca-cert" pod "32d7a3ef-ec4e-4639-b849-baf8078f5fd0" (UID: "32d7a3ef-ec4e-4639-b849-baf8078f5fd0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:56:14.437755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.437729    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-podinfo" pod "32d7a3ef-ec4e-4639-b849-baf8078f5fd0" (UID: "32d7a3ef-ec4e-4639-b849-baf8078f5fd0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:56:14.438002 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.437979    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-envoy" pod "32d7a3ef-ec4e-4639-b849-baf8078f5fd0" (UID: "32d7a3ef-ec4e-4639-b849-baf8078f5fd0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:56:14.438738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.438718    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-token" pod "32d7a3ef-ec4e-4639-b849-baf8078f5fd0" (UID: "32d7a3ef-ec4e-4639-b849-baf8078f5fd0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:56:14.439186 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.439162    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-kube-api-access-n2blw" pod "32d7a3ef-ec4e-4639-b849-baf8078f5fd0" (UID: "32d7a3ef-ec4e-4639-b849-baf8078f5fd0"). InnerVolumeSpecName "kube-api-access-n2blw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:56:14.537841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.537714    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.537841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.537748    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.537841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.537758    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.537841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.537766    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.537841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.537776    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.537841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.537784    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n2blw\" (UniqueName: \"kubernetes.io/projected/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-kube-api-access-n2blw\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.537841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.537795    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.537841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.537803    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32d7a3ef-ec4e-4639-b849-baf8078f5fd0-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:56:14.942620 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.942587    4456 generic.go:356] "Generic (PLEG): container finished" podID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerID="1f3338eaa0ca5d0255f41a33c509df2ba8023c26afcd02725fd1c7f7c3299602" exitCode=0
Nov 02 17:56:14.942848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.942669    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"
Nov 02 17:56:14.942848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.942673    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp" event={"ID":"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e","Type":"ContainerDied","Data":"1f3338eaa0ca5d0255f41a33c509df2ba8023c26afcd02725fd1c7f7c3299602"}
Nov 02 17:56:14.942848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.942727    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp" event={"ID":"a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e","Type":"ContainerDied","Data":"8830dff6831e37d283ecd97e0dc6c3ed4b41844695e566a3d55e22aacc3a9f49"}
Nov 02 17:56:14.942848 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.942749    4456 scope.go:122] "RemoveContainer" containerID="1f3338eaa0ca5d0255f41a33c509df2ba8023c26afcd02725fd1c7f7c3299602"
Nov 02 17:56:14.944602 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.944567    4456 generic.go:356] "Generic (PLEG): container finished" podID="32d7a3ef-ec4e-4639-b849-baf8078f5fd0" containerID="d255c53e82a16333f04bcb424be778697aa2e2fbfb8d5e549007236f80fd5375" exitCode=0
Nov 02 17:56:14.944746 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.944622    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"
Nov 02 17:56:14.944746 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.944700    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql" event={"ID":"32d7a3ef-ec4e-4639-b849-baf8078f5fd0","Type":"ContainerDied","Data":"d255c53e82a16333f04bcb424be778697aa2e2fbfb8d5e549007236f80fd5375"}
Nov 02 17:56:14.944928 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.944755    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql" event={"ID":"32d7a3ef-ec4e-4639-b849-baf8078f5fd0","Type":"ContainerDied","Data":"b198d66cd646f7c5b15d91a4a1e67014c5924d12ab65f8f13eb4759997ff10b4"}
Nov 02 17:56:14.958240 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.958223    4456 scope.go:122] "RemoveContainer" containerID="b53a79d0493a360d98414bc8e408bb9b56548a39e73e6df6cfc4f94d5d63f132"
Nov 02 17:56:14.968498 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.968476    4456 scope.go:122] "RemoveContainer" containerID="1f3338eaa0ca5d0255f41a33c509df2ba8023c26afcd02725fd1c7f7c3299602"
Nov 02 17:56:14.968790 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:56:14.968768    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f3338eaa0ca5d0255f41a33c509df2ba8023c26afcd02725fd1c7f7c3299602\": container with ID starting with 1f3338eaa0ca5d0255f41a33c509df2ba8023c26afcd02725fd1c7f7c3299602 not found: ID does not exist" containerID="1f3338eaa0ca5d0255f41a33c509df2ba8023c26afcd02725fd1c7f7c3299602"
Nov 02 17:56:14.968859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.968799    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f3338eaa0ca5d0255f41a33c509df2ba8023c26afcd02725fd1c7f7c3299602"} err="failed to get container status \"1f3338eaa0ca5d0255f41a33c509df2ba8023c26afcd02725fd1c7f7c3299602\": rpc error: code = NotFound desc = could not find container \"1f3338eaa0ca5d0255f41a33c509df2ba8023c26afcd02725fd1c7f7c3299602\": container with ID starting with 1f3338eaa0ca5d0255f41a33c509df2ba8023c26afcd02725fd1c7f7c3299602 not found: ID does not exist"
Nov 02 17:56:14.968859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.968839    4456 scope.go:122] "RemoveContainer" containerID="b53a79d0493a360d98414bc8e408bb9b56548a39e73e6df6cfc4f94d5d63f132"
Nov 02 17:56:14.969125 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:56:14.969104    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b53a79d0493a360d98414bc8e408bb9b56548a39e73e6df6cfc4f94d5d63f132\": container with ID starting with b53a79d0493a360d98414bc8e408bb9b56548a39e73e6df6cfc4f94d5d63f132 not found: ID does not exist" containerID="b53a79d0493a360d98414bc8e408bb9b56548a39e73e6df6cfc4f94d5d63f132"
Nov 02 17:56:14.969223 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.969135    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b53a79d0493a360d98414bc8e408bb9b56548a39e73e6df6cfc4f94d5d63f132"} err="failed to get container status \"b53a79d0493a360d98414bc8e408bb9b56548a39e73e6df6cfc4f94d5d63f132\": rpc error: code = NotFound desc = could not find container \"b53a79d0493a360d98414bc8e408bb9b56548a39e73e6df6cfc4f94d5d63f132\": container with ID starting with b53a79d0493a360d98414bc8e408bb9b56548a39e73e6df6cfc4f94d5d63f132 not found: ID does not exist"
Nov 02 17:56:14.969223 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.969159    4456 scope.go:122] "RemoveContainer" containerID="d255c53e82a16333f04bcb424be778697aa2e2fbfb8d5e549007236f80fd5375"
Nov 02 17:56:14.979687 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.979669    4456 scope.go:122] "RemoveContainer" containerID="d255c53e82a16333f04bcb424be778697aa2e2fbfb8d5e549007236f80fd5375"
Nov 02 17:56:14.980083 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:56:14.980063    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d255c53e82a16333f04bcb424be778697aa2e2fbfb8d5e549007236f80fd5375\": container with ID starting with d255c53e82a16333f04bcb424be778697aa2e2fbfb8d5e549007236f80fd5375 not found: ID does not exist" containerID="d255c53e82a16333f04bcb424be778697aa2e2fbfb8d5e549007236f80fd5375"
Nov 02 17:56:14.980163 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:14.980088    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d255c53e82a16333f04bcb424be778697aa2e2fbfb8d5e549007236f80fd5375"} err="failed to get container status \"d255c53e82a16333f04bcb424be778697aa2e2fbfb8d5e549007236f80fd5375\": rpc error: code = NotFound desc = could not find container \"d255c53e82a16333f04bcb424be778697aa2e2fbfb8d5e549007236f80fd5375\": container with ID starting with d255c53e82a16333f04bcb424be778697aa2e2fbfb8d5e549007236f80fd5375 not found: ID does not exist"
Nov 02 17:56:15.014614 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:15.014584    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"]
Nov 02 17:56:15.024066 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:15.024037    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-vffbp"]
Nov 02 17:56:15.051464 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:15.051417    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"]
Nov 02 17:56:15.055494 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:15.055467    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-a-c3c112-openshift-default-64b8879ff-n6bql"]
Nov 02 17:56:15.617266 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:15.617236    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32d7a3ef-ec4e-4639-b849-baf8078f5fd0" path="/var/lib/kubelet/pods/32d7a3ef-ec4e-4639-b849-baf8078f5fd0/volumes"
Nov 02 17:56:15.617769 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:15.617755    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" path="/var/lib/kubelet/pods/a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e/volumes"
Nov 02 17:56:15.879718 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:15.879417    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:16.167045 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:56:16.166962    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:56:17.654994 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:17.653363    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:19.501847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:19.500626    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:19.788456 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:19.784701    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:20.384491 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:20.384463    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:20.565987 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:20.565955    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:20.730192 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:20.730155    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:21.624097 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:21.605433    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:22.565880 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:22.563194    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:23.763844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:23.756832    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:23.805076 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:23.803950    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:23.938035 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:23.937965    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:23.979012 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:23.978959    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:24.087060 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:24.086960    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Nov 02 17:56:24.521017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:24.520967    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:24.669026 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:24.668964    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:24.711912 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:24.711870    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 17:56:24.772009 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:24.771901    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:24.814869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:24.804969    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:56:24.814869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:24.805541    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:56:24.877681 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:24.877653    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 17:56:25.630441 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:25.630404    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:27.636738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:27.636672    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:29.384837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:29.383734    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:29.608402 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:29.608231    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:29.876384 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:29.876274    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:30.393165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:30.393085    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:30.574318 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:30.574227    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:31.240104 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:31.240070    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:33.018291 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:33.018255    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:33.314064 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:33.313971    4456 generic.go:356] "Generic (PLEG): container finished" podID="d4161075-440d-4373-b878-57c77612c6d2" containerID="4a6f1801431eae7eaddf5eebcb23a321305fb236e58f980eafe4d90518d86398" exitCode=0
Nov 02 17:56:33.314262 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:33.314050    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" event={"ID":"d4161075-440d-4373-b878-57c77612c6d2","Type":"ContainerDied","Data":"4a6f1801431eae7eaddf5eebcb23a321305fb236e58f980eafe4d90518d86398"}
Nov 02 17:56:33.314662 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:33.314640    4456 scope.go:122] "RemoveContainer" containerID="4a6f1801431eae7eaddf5eebcb23a321305fb236e58f980eafe4d90518d86398"
Nov 02 17:56:33.316689 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:33.316660    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:33.316756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:33.316738    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:33.316796 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:33.316776    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:56:34.327440 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:34.327404    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" event={"ID":"d4161075-440d-4373-b878-57c77612c6d2","Type":"ContainerStarted","Data":"e079ea8bc432c6b15221e1f0f0d1f1094347bece3a5dc356bdb25b749c2da4d5"}
Nov 02 17:56:34.534501 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:34.534466    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:56:34.534800 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:34.534701    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 17:56:34.534970 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:34.534840    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 17:56:34.708840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:34.705383    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:34.865159 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:34.865127    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:35.099730 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:35.099629    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:35.535149 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:35.535112    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 17:56:35.535566 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:35.535175    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 17:56:35.848252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:35.848156    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:36.535197 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:36.535160    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 17:56:36.535655 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:36.535226    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 17:56:36.631528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:36.631496    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:36.920617 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:36.920481    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:37.203570 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:37.203536    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:37.535461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:37.535373    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 17:56:37.535461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:37.535428    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 17:56:38.535572 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:38.535530    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 17:56:38.536118 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:38.535601    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 17:56:38.953182 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:38.953154    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:39.534830 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:39.534791    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused" start-of-body=
Nov 02 17:56:39.535050 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:39.534866    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": dial tcp 10.134.0.168:15021: connect: connection refused"
Nov 02 17:56:40.539936 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:40.539907    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:56:40.540422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:40.540350    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:56:40.541268 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:40.541245    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 17:56:40.631519 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:40.631487    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:42.389758 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:42.389722    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:44.155047 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:44.155017    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:46.089190 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:46.089149    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:46.168071 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:56:46.168037    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:56:47.742674 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:47.742636    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:49.039485 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:49.039441    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"]
Nov 02 17:56:49.051121 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:49.051093    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-b0c4292c-n8sx5"]
Nov 02 17:56:49.618170 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:49.618136    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a8c79b51-5464-4227-b6fc-8a16396aa911" path="/var/lib/kubelet/pods/a8c79b51-5464-4227-b6fc-8a16396aa911/volumes"
Nov 02 17:56:49.777659 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:49.777621    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:51.733783 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:51.731055    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:51.880131 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:51.880097    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:53.161495 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:53.161461    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:53.895877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:53.895832    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:55.661234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:55.661182    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:57.559771 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:57.559738    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:59.404633 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:59.404597    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:56:59.724417 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:56:59.724375    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:01.201827 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:01.201772    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:01.521739 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:01.521583    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:01.931531 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:01.931439    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:02.360357 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.360324    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"]
Nov 02 17:57:02.361138 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.361119    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerName="istio-proxy"
Nov 02 17:57:02.361138 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.361137    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerName="istio-proxy"
Nov 02 17:57:02.361343 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.361189    4456 cpu_manager.go:414] "containerMap: removing container" podUID="34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3" containerName="revoke-keys"
Nov 02 17:57:02.361343 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.361194    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3" containerName="revoke-keys"
Nov 02 17:57:02.361343 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.361205    4456 cpu_manager.go:414] "containerMap: removing container" podUID="32d7a3ef-ec4e-4639-b849-baf8078f5fd0" containerName="istio-proxy"
Nov 02 17:57:02.361343 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.361212    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32d7a3ef-ec4e-4639-b849-baf8078f5fd0" containerName="istio-proxy"
Nov 02 17:57:02.361576 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.361350    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3" containerName="revoke-keys"
Nov 02 17:57:02.361576 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.361367    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="32d7a3ef-ec4e-4639-b849-baf8078f5fd0" containerName="istio-proxy"
Nov 02 17:57:02.361576 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.361383    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerName="istio-proxy"
Nov 02 17:57:02.361576 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.361399    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerName="istio-proxy"
Nov 02 17:57:02.367406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.367385    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.393003 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.392972    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"]
Nov 02 17:57:02.409281 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.409248    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d5b905-openshift-default-dockercfg-vtgjv\""
Nov 02 17:57:02.458003 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.457974    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:02.475783 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.475745    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.476035 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.475799    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rstmc\" (UniqueName: \"kubernetes.io/projected/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-kube-api-access-rstmc\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.476035 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.475908    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.476035 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.475950    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.476035 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.476020    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.476268 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.476070    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.476268 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.476143    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.476268 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.476234    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.476725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.476280    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.578149 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.577968    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.578149 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.578041    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.578149 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.578112    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.578509 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.578189    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.578509 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.578234    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.578509 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.578386    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.578509 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.578416    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rstmc\" (UniqueName: \"kubernetes.io/projected/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-kube-api-access-rstmc\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.610891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.588910    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.610891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.589187    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.610891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.593406    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.610891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.593563    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.610891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.593612    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.610891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.594380    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.610891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.594598    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.636235 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.617596    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rstmc\" (UniqueName: \"kubernetes.io/projected/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-kube-api-access-rstmc\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.637306 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.637099    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.675339 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.673370    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.675339 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.673368    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.695867 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.690127    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:02.794758 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:02.794687    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:03.042506 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:03.042476    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:03.110682 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:03.110658    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"]
Nov 02 17:57:03.111323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:03.111285    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:03.111417 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:03.111376    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:03.111461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:03.111419    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:03.656697 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:03.656662    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" event={"ID":"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21","Type":"ContainerStarted","Data":"026fbcc527da33cce56513069ee46f158e36543f79011d0d0036bc53fe44819c"}
Nov 02 17:57:03.657103 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:03.656713    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" event={"ID":"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21","Type":"ContainerStarted","Data":"e781bf2d2bc7c1a28d8f91cdbfb3c720bee5b614f7581891fa084fcebe1a2d6b"}
Nov 02 17:57:03.706986 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:03.706947    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:03.732666 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:03.732614    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podStartSLOduration=1.7326010379999999 podStartE2EDuration="1.732601038s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:03.717569397 +0000 UTC m=+1640.208632558" watchObservedRunningTime="2026-11-02 17:57:03.732601038 +0000 UTC m=+1640.223664263"
Nov 02 17:57:04.714854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:04.714782    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:04.806124 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:04.806085    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:05.677492 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:05.677456    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:05.678589 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:05.678568    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:57:06.236724 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:06.236687    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:06.409680 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:06.409639    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:06.815244 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:06.815211    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:08.093427 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.093384    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:08.261763 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.261710    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:08.262229 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.262193    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"]
Nov 02 17:57:08.263666 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.263639    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerName="istio-proxy"
Nov 02 17:57:08.263799 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.263786    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f8f4a9-e906-4bb8-a6e6-f985fa71c42e" containerName="istio-proxy"
Nov 02 17:57:08.381153 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.381070    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"]
Nov 02 17:57:08.381481 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.381461    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"
Nov 02 17:57:08.409182 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.409150    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d5b905-dockercfg-sfklt\""
Nov 02 17:57:08.421259 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.416181    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d5b905\""
Nov 02 17:57:08.506412 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.506373    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"]
Nov 02 17:57:08.507495 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.507465    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm" podUID="5f82179e-cc9c-4899-a6fc-1a75c4d5d517" containerName="istio-proxy" containerID="cri-o://b07092894c873f382807fe54f492a1c0a94572b17d28096d4fe738a720dc75c0" gracePeriod=30
Nov 02 17:57:08.569984 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.569889    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr\" (UID: \"e141d845-db9d-4f94-b05e-cfb7848e8fef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"
Nov 02 17:57:08.570161 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.570126    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p9t7z\" (UniqueName: \"kubernetes.io/projected/e141d845-db9d-4f94-b05e-cfb7848e8fef-kube-api-access-p9t7z\") pod \"payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr\" (UID: \"e141d845-db9d-4f94-b05e-cfb7848e8fef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"
Nov 02 17:57:08.616928 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.616892    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:08.671597 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.671510    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr\" (UID: \"e141d845-db9d-4f94-b05e-cfb7848e8fef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"
Nov 02 17:57:08.671764 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.671653    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p9t7z\" (UniqueName: \"kubernetes.io/projected/e141d845-db9d-4f94-b05e-cfb7848e8fef-kube-api-access-p9t7z\") pod \"payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr\" (UID: \"e141d845-db9d-4f94-b05e-cfb7848e8fef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"
Nov 02 17:57:08.672458 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.672434    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr\" (UID: \"e141d845-db9d-4f94-b05e-cfb7848e8fef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"
Nov 02 17:57:08.695893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.695865    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p9t7z\" (UniqueName: \"kubernetes.io/projected/e141d845-db9d-4f94-b05e-cfb7848e8fef-kube-api-access-p9t7z\") pod \"payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr\" (UID: \"e141d845-db9d-4f94-b05e-cfb7848e8fef\") " pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"
Nov 02 17:57:08.778276 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.778240    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:08.803066 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:08.803029    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"
Nov 02 17:57:09.142841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.129475    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"]
Nov 02 17:57:09.160842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.155676    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.199853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.197775    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.199853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.197948    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.199853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.198013    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.199853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.198052    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.199853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.198108    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.199853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.198152    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.199853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.198251    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b8h64\" (UniqueName: \"kubernetes.io/projected/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-kube-api-access-b8h64\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.199853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.198399    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.199853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.198428    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.199853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.199636    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2c9b1d-openshift-default-dockercfg-v7bnw\""
Nov 02 17:57:09.218896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.201281    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"]
Nov 02 17:57:09.281637 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.266412    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:09.300913 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.300448    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.300913 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.300509    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.300913 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.300625    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b8h64\" (UniqueName: \"kubernetes.io/projected/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-kube-api-access-b8h64\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.300913 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.300684    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.300913 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.300714    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.300913 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.300779    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.303674 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.301498    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.303674 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.301577    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.303674 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.301623    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.303674 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.302352    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.303674 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.302576    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.303674 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.303382    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.304622 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.304598    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.308807 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.304841    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.308807 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.308004    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"]
Nov 02 17:57:09.316311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.316240    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.319450 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.319422    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.344252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.342795    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b8h64\" (UniqueName: \"kubernetes.io/projected/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-kube-api-access-b8h64\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.344252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.343262    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.600145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.600105    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:09.722955 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.722912    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" event={"ID":"e141d845-db9d-4f94-b05e-cfb7848e8fef","Type":"ContainerStarted","Data":"723c09d5d9658eb778a71667f74238ad51cacca740ec11432caf9b15357688e9"}
Nov 02 17:57:09.723151 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.722985    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" event={"ID":"e141d845-db9d-4f94-b05e-cfb7848e8fef","Type":"ContainerStarted","Data":"fd67a031eae7babedd49e82fb55f1afbb77cefc2cfe2f850b85471356a865190"}
Nov 02 17:57:09.725718 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.724103    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"
Nov 02 17:57:09.778412 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.761332    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:09.796468 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.786944    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" podStartSLOduration=1.786923912 podStartE2EDuration="1.786923912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:09.774778428 +0000 UTC m=+1646.265841614" watchObservedRunningTime="2026-11-02 17:57:09.786923912 +0000 UTC m=+1646.277987075"
Nov 02 17:57:09.956708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.955876    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"]
Nov 02 17:57:09.957917 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:57:09.957539    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcb09170f_1b12_4a26_8e5b_c5e2c2964a4d.slice/crio-aa577c73e9d0c2359c23f08a768cebf7f5e5f967253d141aa0c98378d78320c0 WatchSource:0}: Error finding container aa577c73e9d0c2359c23f08a768cebf7f5e5f967253d141aa0c98378d78320c0: Status 404 returned error can't find the container with id aa577c73e9d0c2359c23f08a768cebf7f5e5f967253d141aa0c98378d78320c0
Nov 02 17:57:09.960989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.960950    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:09.961119 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.961092    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:09.961201 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:09.961179    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:10.563192 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:10.563162    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:10.733413 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:10.733365    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" event={"ID":"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d","Type":"ContainerStarted","Data":"9d4c1c0786ab2e7395cf39120539ef4b85c634872ec650a8b5de0096e6ff0bd0"}
Nov 02 17:57:10.733596 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:10.733433    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" event={"ID":"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d","Type":"ContainerStarted","Data":"aa577c73e9d0c2359c23f08a768cebf7f5e5f967253d141aa0c98378d78320c0"}
Nov 02 17:57:10.781497 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:10.781443    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podStartSLOduration=1.781428214 podStartE2EDuration="1.781428214s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:10.780036774 +0000 UTC m=+1647.271099948" watchObservedRunningTime="2026-11-02 17:57:10.781428214 +0000 UTC m=+1647.272491375"
Nov 02 17:57:11.601260 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:11.601222    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:11.618685 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:11.618657    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:11.742053 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:11.742023    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:11.743046 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:11.743021    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:57:12.436572 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:12.436533    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:13.037992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.037959    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"]
Nov 02 17:57:13.049527 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.049500    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c4d48df1-xs6xh"]
Nov 02 17:57:13.198623 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.198592    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:13.338029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.337942    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:13.622298 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.622228    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6016b480-9073-45a6-b00b-84d4ab57d67b" path="/var/lib/kubelet/pods/6016b480-9073-45a6-b00b-84d4ab57d67b/volumes"
Nov 02 17:57:13.765427 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.765395    4456 generic.go:356] "Generic (PLEG): container finished" podID="5f82179e-cc9c-4899-a6fc-1a75c4d5d517" containerID="b07092894c873f382807fe54f492a1c0a94572b17d28096d4fe738a720dc75c0" exitCode=0
Nov 02 17:57:13.765577 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.765463    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm" event={"ID":"5f82179e-cc9c-4899-a6fc-1a75c4d5d517","Type":"ContainerDied","Data":"b07092894c873f382807fe54f492a1c0a94572b17d28096d4fe738a720dc75c0"}
Nov 02 17:57:13.797878 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.797853    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:57:13.906281 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.906179    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istiod-ca-cert\") pod \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") "
Nov 02 17:57:13.906281 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.906233    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-workload-certs\") pod \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") "
Nov 02 17:57:13.906547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.906306    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-kube-api-access-qrgxc\" (UniqueName: \"kubernetes.io/projected/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-kube-api-access-qrgxc\") pod \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") "
Nov 02 17:57:13.906547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.906470    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-data\") pod \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") "
Nov 02 17:57:13.906547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.906511    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-workload-socket\") pod \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") "
Nov 02 17:57:13.906726 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.906625    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-workload-certs" pod "5f82179e-cc9c-4899-a6fc-1a75c4d5d517" (UID: "5f82179e-cc9c-4899-a6fc-1a75c4d5d517"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:13.906726 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.906650    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istiod-ca-cert" pod "5f82179e-cc9c-4899-a6fc-1a75c4d5d517" (UID: "5f82179e-cc9c-4899-a6fc-1a75c4d5d517"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:57:13.906726 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.906656    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-credential-socket\") pod \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") "
Nov 02 17:57:13.906918 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.906903    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-data" pod "5f82179e-cc9c-4899-a6fc-1a75c4d5d517" (UID: "5f82179e-cc9c-4899-a6fc-1a75c4d5d517"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:13.906984 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.906917    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-envoy\") pod \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") "
Nov 02 17:57:13.907046 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.907031    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-podinfo\") pod \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") "
Nov 02 17:57:13.907099 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.907060    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-token\" (UniqueName: \"kubernetes.io/projected/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-token\") pod \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\" (UID: \"5f82179e-cc9c-4899-a6fc-1a75c4d5d517\") "
Nov 02 17:57:13.907160 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.906919    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-credential-socket" pod "5f82179e-cc9c-4899-a6fc-1a75c4d5d517" (UID: "5f82179e-cc9c-4899-a6fc-1a75c4d5d517"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:13.907160 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.907128    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-workload-socket" pod "5f82179e-cc9c-4899-a6fc-1a75c4d5d517" (UID: "5f82179e-cc9c-4899-a6fc-1a75c4d5d517"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:13.907860 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.907832    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:13.908024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.907862    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:13.908024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.907878    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:13.908024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.907889    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:13.908024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.907901    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:13.910742 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.910349    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-kube-api-access-qrgxc" pod "5f82179e-cc9c-4899-a6fc-1a75c4d5d517" (UID: "5f82179e-cc9c-4899-a6fc-1a75c4d5d517"). InnerVolumeSpecName "kube-api-access-qrgxc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:13.915983 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.911262    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-podinfo" pod "5f82179e-cc9c-4899-a6fc-1a75c4d5d517" (UID: "5f82179e-cc9c-4899-a6fc-1a75c4d5d517"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:57:13.915983 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.911544    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-token" pod "5f82179e-cc9c-4899-a6fc-1a75c4d5d517" (UID: "5f82179e-cc9c-4899-a6fc-1a75c4d5d517"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:13.915983 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:13.911854    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-envoy" pod "5f82179e-cc9c-4899-a6fc-1a75c4d5d517" (UID: "5f82179e-cc9c-4899-a6fc-1a75c4d5d517"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:14.009654 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:14.009621    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:14.009654 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:14.009652    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:14.009654 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:14.009660    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:14.009928 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:14.009670    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qrgxc\" (UniqueName: \"kubernetes.io/projected/5f82179e-cc9c-4899-a6fc-1a75c4d5d517-kube-api-access-qrgxc\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:14.271595 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:14.271558    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:14.777306 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:14.777278    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"
Nov 02 17:57:14.777499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:14.777319    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm" event={"ID":"5f82179e-cc9c-4899-a6fc-1a75c4d5d517","Type":"ContainerDied","Data":"5d425e10bddf403e64054e56f00f71506fb794b6013c884fae5786249b1a1530"}
Nov 02 17:57:14.777499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:14.777390    4456 scope.go:122] "RemoveContainer" containerID="b07092894c873f382807fe54f492a1c0a94572b17d28096d4fe738a720dc75c0"
Nov 02 17:57:14.845968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:14.845941    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"]
Nov 02 17:57:14.858745 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:14.858712    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-f7zcm"]
Nov 02 17:57:14.953883 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:14.953774    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:15.108755 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.108665    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:15.121453 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.121423    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"]
Nov 02 17:57:15.122624 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.122602    4456 cpu_manager.go:414] "containerMap: removing container" podUID="5f82179e-cc9c-4899-a6fc-1a75c4d5d517" containerName="istio-proxy"
Nov 02 17:57:15.122680 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.122627    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5f82179e-cc9c-4899-a6fc-1a75c4d5d517" containerName="istio-proxy"
Nov 02 17:57:15.122880 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.122867    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5f82179e-cc9c-4899-a6fc-1a75c4d5d517" containerName="istio-proxy"
Nov 02 17:57:15.128666 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.128639    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 17:57:15.145225 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.145195    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2c9b1d-dockercfg-tk2wk\""
Nov 02 17:57:15.146228 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.146195    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2c9b1d\""
Nov 02 17:57:15.152531 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.152196    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"]
Nov 02 17:57:15.217256 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.217189    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"]
Nov 02 17:57:15.225233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.225210    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 17:57:15.235945 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.235610    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn\" (UID: \"3d96e066-10c0-4fe2-9e24-ffd6195b22af\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 17:57:15.235945 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.235784    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rf96w\" (UniqueName: \"kubernetes.io/projected/3d96e066-10c0-4fe2-9e24-ffd6195b22af-kube-api-access-rf96w\") pod \"payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn\" (UID: \"3d96e066-10c0-4fe2-9e24-ffd6195b22af\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 17:57:15.303924 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.303888    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"]
Nov 02 17:57:15.360184 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.352592    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn\" (UID: \"3d96e066-10c0-4fe2-9e24-ffd6195b22af\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 17:57:15.360184 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.352786    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rf96w\" (UniqueName: \"kubernetes.io/projected/3d96e066-10c0-4fe2-9e24-ffd6195b22af-kube-api-access-rf96w\") pod \"payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn\" (UID: \"3d96e066-10c0-4fe2-9e24-ffd6195b22af\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 17:57:15.360184 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.352856    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f7bnx\" (UniqueName: \"kubernetes.io/projected/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-kube-api-access-f7bnx\") pod \"payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6\" (UID: \"60c0e838-5c79-4bf7-a7c6-87903ae8bd8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 17:57:15.360184 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.352963    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6\" (UID: \"60c0e838-5c79-4bf7-a7c6-87903ae8bd8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 17:57:15.360184 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.353727    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn\" (UID: \"3d96e066-10c0-4fe2-9e24-ffd6195b22af\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 17:57:15.391906 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.387675    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rf96w\" (UniqueName: \"kubernetes.io/projected/3d96e066-10c0-4fe2-9e24-ffd6195b22af-kube-api-access-rf96w\") pod \"payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn\" (UID: \"3d96e066-10c0-4fe2-9e24-ffd6195b22af\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 17:57:15.458840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.454896    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f7bnx\" (UniqueName: \"kubernetes.io/projected/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-kube-api-access-f7bnx\") pod \"payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6\" (UID: \"60c0e838-5c79-4bf7-a7c6-87903ae8bd8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 17:57:15.458840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.455044    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6\" (UID: \"60c0e838-5c79-4bf7-a7c6-87903ae8bd8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 17:57:15.458840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.457694    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 17:57:15.458840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.458411    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6\" (UID: \"60c0e838-5c79-4bf7-a7c6-87903ae8bd8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 17:57:15.492027 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.491506    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f7bnx\" (UniqueName: \"kubernetes.io/projected/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-kube-api-access-f7bnx\") pod \"payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6\" (UID: \"60c0e838-5c79-4bf7-a7c6-87903ae8bd8c\") " pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 17:57:15.632595 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.629355    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 17:57:15.679842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:15.662028    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5f82179e-cc9c-4899-a6fc-1a75c4d5d517" path="/var/lib/kubelet/pods/5f82179e-cc9c-4899-a6fc-1a75c4d5d517/volumes"
Nov 02 17:57:16.040557 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:16.040525    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"]
Nov 02 17:57:16.042769 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:57:16.042733    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3d96e066_10c0_4fe2_9e24_ffd6195b22af.slice/crio-d932a52666d4957f9e059f45c9de22ea0c348c2deebf0b105adbc40857774316 WatchSource:0}: Error finding container d932a52666d4957f9e059f45c9de22ea0c348c2deebf0b105adbc40857774316: Status 404 returned error can't find the container with id d932a52666d4957f9e059f45c9de22ea0c348c2deebf0b105adbc40857774316
Nov 02 17:57:16.181872 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:57:16.181792    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:57:16.198134 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:16.198108    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"]
Nov 02 17:57:16.198900 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:57:16.198866    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod60c0e838_5c79_4bf7_a7c6_87903ae8bd8c.slice/crio-c6c35dcca3f6cb5573b9cede13b9a4db8e96921ce5c70e83fd46b5db3efda2a2 WatchSource:0}: Error finding container c6c35dcca3f6cb5573b9cede13b9a4db8e96921ce5c70e83fd46b5db3efda2a2: Status 404 returned error can't find the container with id c6c35dcca3f6cb5573b9cede13b9a4db8e96921ce5c70e83fd46b5db3efda2a2
Nov 02 17:57:16.269203 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:16.269166    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:16.870166 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:16.862682    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" event={"ID":"60c0e838-5c79-4bf7-a7c6-87903ae8bd8c","Type":"ContainerStarted","Data":"5eca9c03a08e76e5e6f070cb8052b78663a3818d87186a38c4402f7bf5034995"}
Nov 02 17:57:16.870166 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:16.862753    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" event={"ID":"60c0e838-5c79-4bf7-a7c6-87903ae8bd8c","Type":"ContainerStarted","Data":"c6c35dcca3f6cb5573b9cede13b9a4db8e96921ce5c70e83fd46b5db3efda2a2"}
Nov 02 17:57:16.870166 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:16.864017    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 17:57:16.870166 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:16.866989    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" event={"ID":"3d96e066-10c0-4fe2-9e24-ffd6195b22af","Type":"ContainerStarted","Data":"4fa143115bc295dd3e06ce938045b94490edee81d1b9aa59b048d0f8a259384b"}
Nov 02 17:57:16.870166 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:16.867038    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" event={"ID":"3d96e066-10c0-4fe2-9e24-ffd6195b22af","Type":"ContainerStarted","Data":"d932a52666d4957f9e059f45c9de22ea0c348c2deebf0b105adbc40857774316"}
Nov 02 17:57:16.870166 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:16.867895    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 17:57:16.923258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:16.923200    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" podStartSLOduration=1.923184437 podStartE2EDuration="1.923184437s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:16.908485255 +0000 UTC m=+1653.399548417" watchObservedRunningTime="2026-11-02 17:57:16.923184437 +0000 UTC m=+1653.414247598"
Nov 02 17:57:16.970461 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:16.970399    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" podStartSLOduration=1.970383518 podStartE2EDuration="1.970383518s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:15 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:16.966251652 +0000 UTC m=+1653.457314813" watchObservedRunningTime="2026-11-02 17:57:16.970383518 +0000 UTC m=+1653.461446678"
Nov 02 17:57:18.125854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:18.121683    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:18.277404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:18.277346    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:18.472553 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:18.472519    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:20.199522 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:20.199483    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:21.561144 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:21.561116    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:21.788838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:21.788469    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"
Nov 02 17:57:22.037852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:22.037106    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:23.844716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:23.844660    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:25.512066 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.511982    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"]
Nov 02 17:57:25.594093 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.594049    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:25.623307 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.622225    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"
Nov 02 17:57:25.638700 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.638670    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 17:57:25.640482 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.640456    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 17:57:25.643992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.643967    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"]
Nov 02 17:57:25.667281 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.667249    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ldckl\" (UniqueName: \"kubernetes.io/projected/a75b25ed-ab5b-40e4-8305-59048777fdac-kube-api-access-ldckl\") pod \"maas-api-revoke-sub-ed264d03-vc2hg\" (UID: \"a75b25ed-ab5b-40e4-8305-59048777fdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"
Nov 02 17:57:25.667494 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.667384    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a75b25ed-ab5b-40e4-8305-59048777fdac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed264d03-vc2hg\" (UID: \"a75b25ed-ab5b-40e4-8305-59048777fdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"
Nov 02 17:57:25.667494 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.667461    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a75b25ed-ab5b-40e4-8305-59048777fdac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed264d03-vc2hg\" (UID: \"a75b25ed-ab5b-40e4-8305-59048777fdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"
Nov 02 17:57:25.769061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.768977    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ldckl\" (UniqueName: \"kubernetes.io/projected/a75b25ed-ab5b-40e4-8305-59048777fdac-kube-api-access-ldckl\") pod \"maas-api-revoke-sub-ed264d03-vc2hg\" (UID: \"a75b25ed-ab5b-40e4-8305-59048777fdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"
Nov 02 17:57:25.769348 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.769108    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a75b25ed-ab5b-40e4-8305-59048777fdac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed264d03-vc2hg\" (UID: \"a75b25ed-ab5b-40e4-8305-59048777fdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"
Nov 02 17:57:25.769348 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.769177    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a75b25ed-ab5b-40e4-8305-59048777fdac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed264d03-vc2hg\" (UID: \"a75b25ed-ab5b-40e4-8305-59048777fdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"
Nov 02 17:57:25.769992 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.769923    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a75b25ed-ab5b-40e4-8305-59048777fdac-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ed264d03-vc2hg\" (UID: \"a75b25ed-ab5b-40e4-8305-59048777fdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"
Nov 02 17:57:25.771740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.771718    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a75b25ed-ab5b-40e4-8305-59048777fdac-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ed264d03-vc2hg\" (UID: \"a75b25ed-ab5b-40e4-8305-59048777fdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"
Nov 02 17:57:25.801090 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.801058    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ldckl\" (UniqueName: \"kubernetes.io/projected/a75b25ed-ab5b-40e4-8305-59048777fdac-kube-api-access-ldckl\") pod \"maas-api-revoke-sub-ed264d03-vc2hg\" (UID: \"a75b25ed-ab5b-40e4-8305-59048777fdac\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"
Nov 02 17:57:25.988960 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:25.988928    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"
Nov 02 17:57:26.356043 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:57:26.356008    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda75b25ed_ab5b_40e4_8305_59048777fdac.slice/crio-178dba41da1cee1dc9e0ce5cdde6d8e6f8fa0e0e90a84cf6a6c2baeb41d8236b WatchSource:0}: Error finding container 178dba41da1cee1dc9e0ce5cdde6d8e6f8fa0e0e90a84cf6a6c2baeb41d8236b: Status 404 returned error can't find the container with id 178dba41da1cee1dc9e0ce5cdde6d8e6f8fa0e0e90a84cf6a6c2baeb41d8236b
Nov 02 17:57:26.356301 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:26.356283    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"]
Nov 02 17:57:27.096427 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:27.096388    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg" event={"ID":"a75b25ed-ab5b-40e4-8305-59048777fdac","Type":"ContainerStarted","Data":"e1922175d5513058013bf3b85e2a3221900f1fde3d1509b5419561d2ef700c83"}
Nov 02 17:57:27.096942 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:27.096447    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg" event={"ID":"a75b25ed-ab5b-40e4-8305-59048777fdac","Type":"ContainerStarted","Data":"178dba41da1cee1dc9e0ce5cdde6d8e6f8fa0e0e90a84cf6a6c2baeb41d8236b"}
Nov 02 17:57:27.145312 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:27.145254    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg" podStartSLOduration=2.145218514 podStartE2EDuration="2.145218514s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:25 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:27.139147072 +0000 UTC m=+1663.630210258" watchObservedRunningTime="2026-11-02 17:57:27.145218514 +0000 UTC m=+1663.636281676"
Nov 02 17:57:27.342701 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:27.342621    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:28.107401 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:28.107304    4456 generic.go:356] "Generic (PLEG): container finished" podID="a75b25ed-ab5b-40e4-8305-59048777fdac" containerID="e1922175d5513058013bf3b85e2a3221900f1fde3d1509b5419561d2ef700c83" exitCode=0
Nov 02 17:57:28.108000 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:28.107391    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg" event={"ID":"a75b25ed-ab5b-40e4-8305-59048777fdac","Type":"ContainerDied","Data":"e1922175d5513058013bf3b85e2a3221900f1fde3d1509b5419561d2ef700c83"}
Nov 02 17:57:28.962329 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:28.962288    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 17:57:28.962749 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:28.962729    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 17:57:29.121002 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.120913    4456 generic.go:356] "Generic (PLEG): container finished" podID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerID="0b5a297c1cf1ce7307713189a5bb582c30e4186d062817fc68e564f7bf4411ba" exitCode=0
Nov 02 17:57:29.121359 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.120995    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" event={"ID":"82ba53b4-e91a-4299-9498-f5abcd34576b","Type":"ContainerDied","Data":"0b5a297c1cf1ce7307713189a5bb582c30e4186d062817fc68e564f7bf4411ba"}
Nov 02 17:57:29.122166 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.121579    4456 scope.go:122] "RemoveContainer" containerID="0b5a297c1cf1ce7307713189a5bb582c30e4186d062817fc68e564f7bf4411ba"
Nov 02 17:57:29.124029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.123997    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:29.124172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.124086    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:29.124172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.124124    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:29.133169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.133148    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:29.426233 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.426205    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"
Nov 02 17:57:29.546538 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.546480    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a75b25ed-ab5b-40e4-8305-59048777fdac-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a75b25ed-ab5b-40e4-8305-59048777fdac-maas-api-xks-ca\") pod \"a75b25ed-ab5b-40e4-8305-59048777fdac\" (UID: \"a75b25ed-ab5b-40e4-8305-59048777fdac\") "
Nov 02 17:57:29.546730 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.546617    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a75b25ed-ab5b-40e4-8305-59048777fdac-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a75b25ed-ab5b-40e4-8305-59048777fdac-maas-api-openshift-ca\") pod \"a75b25ed-ab5b-40e4-8305-59048777fdac\" (UID: \"a75b25ed-ab5b-40e4-8305-59048777fdac\") "
Nov 02 17:57:29.546990 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.546965    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a75b25ed-ab5b-40e4-8305-59048777fdac-kube-api-access-ldckl\" (UniqueName: \"kubernetes.io/projected/a75b25ed-ab5b-40e4-8305-59048777fdac-kube-api-access-ldckl\") pod \"a75b25ed-ab5b-40e4-8305-59048777fdac\" (UID: \"a75b25ed-ab5b-40e4-8305-59048777fdac\") "
Nov 02 17:57:29.547760 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.547730    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a75b25ed-ab5b-40e4-8305-59048777fdac-maas-api-openshift-ca" pod "a75b25ed-ab5b-40e4-8305-59048777fdac" (UID: "a75b25ed-ab5b-40e4-8305-59048777fdac"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:57:29.549306 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.549278    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a75b25ed-ab5b-40e4-8305-59048777fdac-maas-api-xks-ca" pod "a75b25ed-ab5b-40e4-8305-59048777fdac" (UID: "a75b25ed-ab5b-40e4-8305-59048777fdac"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 17:57:29.550036 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.550013    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a75b25ed-ab5b-40e4-8305-59048777fdac-kube-api-access-ldckl" pod "a75b25ed-ab5b-40e4-8305-59048777fdac" (UID: "a75b25ed-ab5b-40e4-8305-59048777fdac"). InnerVolumeSpecName "kube-api-access-ldckl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:29.649165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.649095    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ldckl\" (UniqueName: \"kubernetes.io/projected/a75b25ed-ab5b-40e4-8305-59048777fdac-kube-api-access-ldckl\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:29.649165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.649121    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a75b25ed-ab5b-40e4-8305-59048777fdac-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:29.649165 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:29.649133    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a75b25ed-ab5b-40e4-8305-59048777fdac-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:30.133958 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:30.133928    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"
Nov 02 17:57:30.134422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:30.133961    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg" event={"ID":"a75b25ed-ab5b-40e4-8305-59048777fdac","Type":"ContainerDied","Data":"178dba41da1cee1dc9e0ce5cdde6d8e6f8fa0e0e90a84cf6a6c2baeb41d8236b"}
Nov 02 17:57:30.134422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:30.134025    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="178dba41da1cee1dc9e0ce5cdde6d8e6f8fa0e0e90a84cf6a6c2baeb41d8236b"
Nov 02 17:57:30.136339 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:30.136309    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" event={"ID":"82ba53b4-e91a-4299-9498-f5abcd34576b","Type":"ContainerStarted","Data":"e0bbfd6f9c197b1322c631c19d97f927c7c4b56631d9cd54e8dd5474d3b87b5d"}
Nov 02 17:57:30.595652 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:30.595619    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:57:30.596377 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:30.596055    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 17:57:30.596377 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:30.596103    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 17:57:30.847228 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:30.847137    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:31.597051 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:31.597011    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 17:57:31.597667 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:31.597634    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 17:57:32.613843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:32.613570    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 17:57:32.613843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:32.613634    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 17:57:32.618592 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:32.618558    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:33.612895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:33.602866    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 17:57:33.612895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:33.602933    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 17:57:33.942877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:33.942840    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"]
Nov 02 17:57:33.943387 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:33.943093    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96" podUID="3891e06c-0c53-421c-b379-c38efca65905" containerName="istio-proxy" containerID="cri-o://fb40301125fc84beda005c90f6eecf66143cedc0c31f6edeedcdf53df1b482b3" gracePeriod=30
Nov 02 17:57:34.417363 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:34.417268    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:34.610844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:34.608835    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 17:57:34.610844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:34.608905    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 17:57:35.420146 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:35.420104    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:57:35.588216 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:35.588173    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused" start-of-body=
Nov 02 17:57:35.588392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:35.588260    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": dial tcp 10.134.0.174:15021: connect: connection refused"
Nov 02 17:57:35.976397 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:35.976365    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:36.312137 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.312045    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:36.515205 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.515161    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:36.592759 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.592685    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:57:36.593635 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.593615    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 17:57:36.825253 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.825216    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"]
Nov 02 17:57:36.826464 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.826436    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a75b25ed-ab5b-40e4-8305-59048777fdac" containerName="revoke-keys"
Nov 02 17:57:36.826464 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.826462    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a75b25ed-ab5b-40e4-8305-59048777fdac" containerName="revoke-keys"
Nov 02 17:57:36.826700 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.826686    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a75b25ed-ab5b-40e4-8305-59048777fdac" containerName="revoke-keys"
Nov 02 17:57:36.832775 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.832757    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:36.848460 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.848401    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-96768b-openshift-default-dockercfg-v8qql\""
Nov 02 17:57:36.871422 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.871396    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"]
Nov 02 17:57:36.892204 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.892168    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:36.892346 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.892268    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:36.892405 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.892363    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:36.892521 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.892504    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:36.892635 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.892602    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:36.892712 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.892694    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:36.892790 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.892777    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:36.892889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.892874    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6vtvv\" (UniqueName: \"kubernetes.io/projected/67b961eb-ab19-4d8d-bb2a-db7c94a93436-kube-api-access-6vtvv\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:36.893003 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:36.892925    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.006840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.005961    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.006840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.006040    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6vtvv\" (UniqueName: \"kubernetes.io/projected/67b961eb-ab19-4d8d-bb2a-db7c94a93436-kube-api-access-6vtvv\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.006840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.006098    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.006840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.006172    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.006840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.006216    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.006840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.006254    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.006840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.006398    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.006840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.006490    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.006840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.006576    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.007487 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.007074    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.022921 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.008108    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.022921 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.008663    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.022921 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.009164    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.022921 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.009385    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.053315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.053275    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:37.078490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.078457    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.079029 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.078975    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.082058 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.082034    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.083019 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.082968    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6vtvv\" (UniqueName: \"kubernetes.io/projected/67b961eb-ab19-4d8d-bb2a-db7c94a93436-kube-api-access-6vtvv\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.146421 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.146334    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:37.446591 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.446550    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:37.528715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.528684    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"]
Nov 02 17:57:37.529602 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:57:37.529558    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67b961eb_ab19_4d8d_bb2a_db7c94a93436.slice/crio-42ee0fddf775b9d2d49f7e881c391a9dbd7969d720ee7c45060a8dbe249c0fae WatchSource:0}: Error finding container 42ee0fddf775b9d2d49f7e881c391a9dbd7969d720ee7c45060a8dbe249c0fae: Status 404 returned error can't find the container with id 42ee0fddf775b9d2d49f7e881c391a9dbd7969d720ee7c45060a8dbe249c0fae
Nov 02 17:57:37.533630 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.533588    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:37.533765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.533680    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:37.533765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:37.533727    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:38.269189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:38.269152    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" event={"ID":"67b961eb-ab19-4d8d-bb2a-db7c94a93436","Type":"ContainerStarted","Data":"17526f5191570a9cb7a3efb0f2691ee4078b3c90dee5e66f8d2131400d0a1ec8"}
Nov 02 17:57:38.269404 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:38.269208    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" event={"ID":"67b961eb-ab19-4d8d-bb2a-db7c94a93436","Type":"ContainerStarted","Data":"42ee0fddf775b9d2d49f7e881c391a9dbd7969d720ee7c45060a8dbe249c0fae"}
Nov 02 17:57:38.319488 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:38.319431    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podStartSLOduration=2.319416866 podStartE2EDuration="2.319416866s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:38.315454835 +0000 UTC m=+1674.806518024" watchObservedRunningTime="2026-11-02 17:57:38.319416866 +0000 UTC m=+1674.810480027"
Nov 02 17:57:38.334921 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:38.334837    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:39.147256 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.147134    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:39.156271 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.155166    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:39.248264 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.248236    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:57:39.279319 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.279279    4456 generic.go:356] "Generic (PLEG): container finished" podID="3891e06c-0c53-421c-b379-c38efca65905" containerID="fb40301125fc84beda005c90f6eecf66143cedc0c31f6edeedcdf53df1b482b3" exitCode=0
Nov 02 17:57:39.279499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.279333    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"
Nov 02 17:57:39.279499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.279359    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96" event={"ID":"3891e06c-0c53-421c-b379-c38efca65905","Type":"ContainerDied","Data":"fb40301125fc84beda005c90f6eecf66143cedc0c31f6edeedcdf53df1b482b3"}
Nov 02 17:57:39.279499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.279414    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96" event={"ID":"3891e06c-0c53-421c-b379-c38efca65905","Type":"ContainerDied","Data":"2318d551484065b666e190c0c034c38972b8723fb968f8039c1421518bc39ab1"}
Nov 02 17:57:39.279499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.279441    4456 scope.go:122] "RemoveContainer" containerID="fb40301125fc84beda005c90f6eecf66143cedc0c31f6edeedcdf53df1b482b3"
Nov 02 17:57:39.279879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.279789    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:39.281054 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.281035    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:57:39.291646 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.291630    4456 scope.go:122] "RemoveContainer" containerID="fb40301125fc84beda005c90f6eecf66143cedc0c31f6edeedcdf53df1b482b3"
Nov 02 17:57:39.291937 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:57:39.291918    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb40301125fc84beda005c90f6eecf66143cedc0c31f6edeedcdf53df1b482b3\": container with ID starting with fb40301125fc84beda005c90f6eecf66143cedc0c31f6edeedcdf53df1b482b3 not found: ID does not exist" containerID="fb40301125fc84beda005c90f6eecf66143cedc0c31f6edeedcdf53df1b482b3"
Nov 02 17:57:39.292009 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.291943    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb40301125fc84beda005c90f6eecf66143cedc0c31f6edeedcdf53df1b482b3"} err="failed to get container status \"fb40301125fc84beda005c90f6eecf66143cedc0c31f6edeedcdf53df1b482b3\": rpc error: code = NotFound desc = could not find container \"fb40301125fc84beda005c90f6eecf66143cedc0c31f6edeedcdf53df1b482b3\": container with ID starting with fb40301125fc84beda005c90f6eecf66143cedc0c31f6edeedcdf53df1b482b3 not found: ID does not exist"
Nov 02 17:57:39.363202 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.363144    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-credential-socket\") pod \"3891e06c-0c53-421c-b379-c38efca65905\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") "
Nov 02 17:57:39.363425 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.363306    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3891e06c-0c53-421c-b379-c38efca65905-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3891e06c-0c53-421c-b379-c38efca65905-istiod-ca-cert\") pod \"3891e06c-0c53-421c-b379-c38efca65905\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") "
Nov 02 17:57:39.363425 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.363350    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3891e06c-0c53-421c-b379-c38efca65905-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3891e06c-0c53-421c-b379-c38efca65905-istio-podinfo\") pod \"3891e06c-0c53-421c-b379-c38efca65905\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") "
Nov 02 17:57:39.363425 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.363410    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3891e06c-0c53-421c-b379-c38efca65905-kube-api-access-btsdk\" (UniqueName: \"kubernetes.io/projected/3891e06c-0c53-421c-b379-c38efca65905-kube-api-access-btsdk\") pod \"3891e06c-0c53-421c-b379-c38efca65905\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") "
Nov 02 17:57:39.363632 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.363486    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-workload-socket\") pod \"3891e06c-0c53-421c-b379-c38efca65905\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") "
Nov 02 17:57:39.364006 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.363976    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-workload-certs\") pod \"3891e06c-0c53-421c-b379-c38efca65905\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") "
Nov 02 17:57:39.364151 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.364095    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-istio-data\") pod \"3891e06c-0c53-421c-b379-c38efca65905\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") "
Nov 02 17:57:39.364252 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.364229    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3891e06c-0c53-421c-b379-c38efca65905-istio-token\" (UniqueName: \"kubernetes.io/projected/3891e06c-0c53-421c-b379-c38efca65905-istio-token\") pod \"3891e06c-0c53-421c-b379-c38efca65905\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") "
Nov 02 17:57:39.364373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.364270    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-istio-envoy\") pod \"3891e06c-0c53-421c-b379-c38efca65905\" (UID: \"3891e06c-0c53-421c-b379-c38efca65905\") "
Nov 02 17:57:39.364373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.364306    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-credential-socket" pod "3891e06c-0c53-421c-b379-c38efca65905" (UID: "3891e06c-0c53-421c-b379-c38efca65905"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:39.364634 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.364535    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3891e06c-0c53-421c-b379-c38efca65905-istiod-ca-cert" pod "3891e06c-0c53-421c-b379-c38efca65905" (UID: "3891e06c-0c53-421c-b379-c38efca65905"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:57:39.365993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.365760    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:39.365993 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.365790    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3891e06c-0c53-421c-b379-c38efca65905-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:39.368132 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.368090    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-workload-certs" pod "3891e06c-0c53-421c-b379-c38efca65905" (UID: "3891e06c-0c53-421c-b379-c38efca65905"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:39.368254 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.367888    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-istio-data" pod "3891e06c-0c53-421c-b379-c38efca65905" (UID: "3891e06c-0c53-421c-b379-c38efca65905"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:39.372949 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.369989    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-istio-envoy" pod "3891e06c-0c53-421c-b379-c38efca65905" (UID: "3891e06c-0c53-421c-b379-c38efca65905"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:39.380158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.380134    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3891e06c-0c53-421c-b379-c38efca65905-istio-podinfo" pod "3891e06c-0c53-421c-b379-c38efca65905" (UID: "3891e06c-0c53-421c-b379-c38efca65905"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:57:39.380704 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.380661    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3891e06c-0c53-421c-b379-c38efca65905-istio-token" pod "3891e06c-0c53-421c-b379-c38efca65905" (UID: "3891e06c-0c53-421c-b379-c38efca65905"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:39.382944 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.382920    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-workload-socket" pod "3891e06c-0c53-421c-b379-c38efca65905" (UID: "3891e06c-0c53-421c-b379-c38efca65905"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:39.393891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.387874    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3891e06c-0c53-421c-b379-c38efca65905-kube-api-access-btsdk" pod "3891e06c-0c53-421c-b379-c38efca65905" (UID: "3891e06c-0c53-421c-b379-c38efca65905"). InnerVolumeSpecName "kube-api-access-btsdk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:39.467476 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.467443    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:39.467476 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.467476    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3891e06c-0c53-421c-b379-c38efca65905-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:39.467669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.467491    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:39.467669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.467504    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3891e06c-0c53-421c-b379-c38efca65905-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:39.467669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.467518    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-btsdk\" (UniqueName: \"kubernetes.io/projected/3891e06c-0c53-421c-b379-c38efca65905-kube-api-access-btsdk\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:39.467669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.467532    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:39.467669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.467545    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3891e06c-0c53-421c-b379-c38efca65905-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:39.629560 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.629531    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"]
Nov 02 17:57:39.642248 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.642220    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-6ct96"]
Nov 02 17:57:39.818618 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:39.818583    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:40.612315 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:40.612279    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:40.719725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:40.705706    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"]
Nov 02 17:57:40.719725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:40.719623    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-api-b-227833-z5rnk"]
Nov 02 17:57:40.788575 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:40.788524    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:40.989471 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:40.989439    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:41.147707 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:41.147673    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:41.410708 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:41.410625    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:41.618020 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:41.617989    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3" path="/var/lib/kubelet/pods/34c4fc47-fe3b-4fa1-89ca-dcae6a7950c3/volumes"
Nov 02 17:57:41.618605 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:41.618588    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3891e06c-0c53-421c-b379-c38efca65905" path="/var/lib/kubelet/pods/3891e06c-0c53-421c-b379-c38efca65905/volumes"
Nov 02 17:57:42.725235 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:42.725198    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:42.877920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:42.877885    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:42.881977 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:42.881952    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"]
Nov 02 17:57:42.882706 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:42.882692    4456 cpu_manager.go:414] "containerMap: removing container" podUID="3891e06c-0c53-421c-b379-c38efca65905" containerName="istio-proxy"
Nov 02 17:57:42.882759 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:42.882708    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3891e06c-0c53-421c-b379-c38efca65905" containerName="istio-proxy"
Nov 02 17:57:42.882882 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:42.882872    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="3891e06c-0c53-421c-b379-c38efca65905" containerName="istio-proxy"
Nov 02 17:57:42.888285 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:42.888268    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"
Nov 02 17:57:42.906803 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:42.906775    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"]
Nov 02 17:57:42.911158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:42.911132    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-96768b-dockercfg-dm54d\""
Nov 02 17:57:42.914998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:42.914956    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-96768b\""
Nov 02 17:57:42.977887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:42.970455    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k85vn\" (UniqueName: \"kubernetes.io/projected/22b556c1-112b-49d4-932c-0143e912a54e-kube-api-access-k85vn\") pod \"payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr\" (UID: \"22b556c1-112b-49d4-932c-0143e912a54e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"
Nov 02 17:57:42.977887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:42.970932    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr\" (UID: \"22b556c1-112b-49d4-932c-0143e912a54e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"
Nov 02 17:57:43.035893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.034295    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:43.044835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.038891    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"]
Nov 02 17:57:43.111841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.101665    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr\" (UID: \"22b556c1-112b-49d4-932c-0143e912a54e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"
Nov 02 17:57:43.111841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.101792    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k85vn\" (UniqueName: \"kubernetes.io/projected/22b556c1-112b-49d4-932c-0143e912a54e-kube-api-access-k85vn\") pod \"payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr\" (UID: \"22b556c1-112b-49d4-932c-0143e912a54e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"
Nov 02 17:57:43.197973 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.195900    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr\" (UID: \"22b556c1-112b-49d4-932c-0143e912a54e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"
Nov 02 17:57:43.222838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.212682    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"
Nov 02 17:57:43.232895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.230889    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"]
Nov 02 17:57:43.248048 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.248010    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k85vn\" (UniqueName: \"kubernetes.io/projected/22b556c1-112b-49d4-932c-0143e912a54e-kube-api-access-k85vn\") pod \"payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr\" (UID: \"22b556c1-112b-49d4-932c-0143e912a54e\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"
Nov 02 17:57:43.409485 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.409450    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f\" (UID: \"5ce3335c-36e6-4018-a264-5b79085f09f3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"
Nov 02 17:57:43.409485 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.409489    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ct2fq\" (UniqueName: \"kubernetes.io/projected/5ce3335c-36e6-4018-a264-5b79085f09f3-kube-api-access-ct2fq\") pod \"payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f\" (UID: \"5ce3335c-36e6-4018-a264-5b79085f09f3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"
Nov 02 17:57:43.503998 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.503916    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"
Nov 02 17:57:43.510880 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.510852    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f\" (UID: \"5ce3335c-36e6-4018-a264-5b79085f09f3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"
Nov 02 17:57:43.511010 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.510889    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ct2fq\" (UniqueName: \"kubernetes.io/projected/5ce3335c-36e6-4018-a264-5b79085f09f3-kube-api-access-ct2fq\") pod \"payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f\" (UID: \"5ce3335c-36e6-4018-a264-5b79085f09f3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"
Nov 02 17:57:43.511587 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.511564    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f\" (UID: \"5ce3335c-36e6-4018-a264-5b79085f09f3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"
Nov 02 17:57:43.536982 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.536831    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ct2fq\" (UniqueName: \"kubernetes.io/projected/5ce3335c-36e6-4018-a264-5b79085f09f3-kube-api-access-ct2fq\") pod \"payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f\" (UID: \"5ce3335c-36e6-4018-a264-5b79085f09f3\") " pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"
Nov 02 17:57:43.578157 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.576067    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"
Nov 02 17:57:43.840971 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:57:43.840942    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod22b556c1_112b_49d4_932c_0143e912a54e.slice/crio-73ad40a2e08e2a89f3f3a65b0278695bf804ddfca38bfd4dc0b160b370ad7f6a WatchSource:0}: Error finding container 73ad40a2e08e2a89f3f3a65b0278695bf804ddfca38bfd4dc0b160b370ad7f6a: Status 404 returned error can't find the container with id 73ad40a2e08e2a89f3f3a65b0278695bf804ddfca38bfd4dc0b160b370ad7f6a
Nov 02 17:57:43.841342 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.841261    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"]
Nov 02 17:57:43.963515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:43.963474    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"]
Nov 02 17:57:43.963765 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:57:43.963742    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5ce3335c_36e6_4018_a264_5b79085f09f3.slice/crio-928520dc82369ea603f9a3673bc51eaa21c7120c28287ee57759d659ede44319 WatchSource:0}: Error finding container 928520dc82369ea603f9a3673bc51eaa21c7120c28287ee57759d659ede44319: Status 404 returned error can't find the container with id 928520dc82369ea603f9a3673bc51eaa21c7120c28287ee57759d659ede44319
Nov 02 17:57:44.374503 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:44.374452    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" event={"ID":"22b556c1-112b-49d4-932c-0143e912a54e","Type":"ContainerStarted","Data":"968e2de3ae2524b2141ebfc9d2cc91b4232330bb2b1e8308c1e4e1329d68b932"}
Nov 02 17:57:44.374693 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:44.374565    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"
Nov 02 17:57:44.374693 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:44.374590    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" event={"ID":"22b556c1-112b-49d4-932c-0143e912a54e","Type":"ContainerStarted","Data":"73ad40a2e08e2a89f3f3a65b0278695bf804ddfca38bfd4dc0b160b370ad7f6a"}
Nov 02 17:57:44.376280 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:44.376254    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" event={"ID":"5ce3335c-36e6-4018-a264-5b79085f09f3","Type":"ContainerStarted","Data":"0dcde13a6a52ff4eff6e292ac044743343cbb0afa2e3f9446f8189685a9bd665"}
Nov 02 17:57:44.376392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:44.376297    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" event={"ID":"5ce3335c-36e6-4018-a264-5b79085f09f3","Type":"ContainerStarted","Data":"928520dc82369ea603f9a3673bc51eaa21c7120c28287ee57759d659ede44319"}
Nov 02 17:57:44.376392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:44.376337    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"
Nov 02 17:57:44.425792 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:44.425734    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" podStartSLOduration=2.425718496 podStartE2EDuration="2.425718496s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:44.422547421 +0000 UTC m=+1680.913610595" watchObservedRunningTime="2026-11-02 17:57:44.425718496 +0000 UTC m=+1680.916781656"
Nov 02 17:57:44.485391 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:44.485331    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" podStartSLOduration=2.485313422 podStartE2EDuration="2.485313422s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:42 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:44.483513769 +0000 UTC m=+1680.974576932" watchObservedRunningTime="2026-11-02 17:57:44.485313422 +0000 UTC m=+1680.976376586"
Nov 02 17:57:44.981402 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:44.981002    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:46.166939 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:57:46.166901    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:57:46.409214 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:46.409184    4456 scope.go:122] "RemoveContainer" containerID="d5a7f9bff5a598255785a777f7ede52ce5db84493e91ffbd489bc8aa448220fc"
Nov 02 17:57:46.422197 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:46.422170    4456 scope.go:122] "RemoveContainer" containerID="9a0a559a6845e4aa2c0fa679f058981c14eda9f74bdce3b25cbc0a31d3b5fa98"
Nov 02 17:57:46.433193 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:46.433169    4456 scope.go:122] "RemoveContainer" containerID="b7b1ff2346b373c8a5ff7b17629bb45aecfe81aa4915e82fb8d4cb26884b87a7"
Nov 02 17:57:46.803008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:46.802972    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:47.972230 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:47.972193    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:48.141903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:48.141867    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:48.341200 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:48.341118    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"]
Nov 02 17:57:48.341430 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:48.341383    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy" containerID="cri-o://54ec2269cb5bd8ac76efbd725adbaeb88ab8284252404474208d847effc04e1b" gracePeriod=30
Nov 02 17:57:48.448875 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:48.444582    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:48.719237 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:48.719179    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:49.249598 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.249556    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:49.636499 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.636414    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"]
Nov 02 17:57:49.643429 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.643401    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.668035 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.668005    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"]
Nov 02 17:57:49.669770 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.669743    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-872354-openshift-default-dockercfg-rdxd7\""
Nov 02 17:57:49.746563 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.746518    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-istio-data\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.826501 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.826458    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-istio-envoy\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.827477 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.827446    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32802945-5e87-4339-a7eb-33212aab7ba5-istio-podinfo\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.827963 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.827942    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-workload-certs\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.828488 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.828471    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32802945-5e87-4339-a7eb-33212aab7ba5-istio-token\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.828889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.828870    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbr2n\" (UniqueName: \"kubernetes.io/projected/32802945-5e87-4339-a7eb-33212aab7ba5-kube-api-access-qbr2n\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.829056 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.829039    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-credential-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.829711 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.829694    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32802945-5e87-4339-a7eb-33212aab7ba5-istiod-ca-cert\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.829975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.829957    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-workload-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.871322 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.871231    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:49.939836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.936654    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32802945-5e87-4339-a7eb-33212aab7ba5-istiod-ca-cert\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.939836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.936732    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-workload-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.939836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.936762    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-istio-data\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.939836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.936897    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-istio-envoy\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.939836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.937041    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32802945-5e87-4339-a7eb-33212aab7ba5-istio-podinfo\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.939836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.937122    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-workload-certs\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.939836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.937223    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32802945-5e87-4339-a7eb-33212aab7ba5-istio-token\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.939836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.937262    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbr2n\" (UniqueName: \"kubernetes.io/projected/32802945-5e87-4339-a7eb-33212aab7ba5-kube-api-access-qbr2n\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.939836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.937299    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-credential-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.939836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.937839    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-credential-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.939836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.939163    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32802945-5e87-4339-a7eb-33212aab7ba5-istiod-ca-cert\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.939836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.939404    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-workload-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.939836 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.939648    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-istio-data\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.951895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.945020    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-workload-certs\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.955873 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.954389    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32802945-5e87-4339-a7eb-33212aab7ba5-istio-podinfo\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.955873 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.955065    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-istio-envoy\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:49.987839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:49.982346    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbr2n\" (UniqueName: \"kubernetes.io/projected/32802945-5e87-4339-a7eb-33212aab7ba5-kube-api-access-qbr2n\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:50.017834 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:50.017546    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32802945-5e87-4339-a7eb-33212aab7ba5-istio-token\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-v5qhw\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:50.240546 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:50.239115    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:50.264876 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:50.264843    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:50.615518 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:50.615490    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"]
Nov 02 17:57:50.617857 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:50.617804    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:50.617988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:50.617906    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:50.617988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:50.617949    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:57:50.816841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:50.816303    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:51.443277 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:51.443240    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" event={"ID":"32802945-5e87-4339-a7eb-33212aab7ba5","Type":"ContainerStarted","Data":"be2b1174a81ac8ec0f45fb11bddb3dc43efc0e67214ba6610f1874b6ec7eb231"}
Nov 02 17:57:51.443703 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:51.443292    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" event={"ID":"32802945-5e87-4339-a7eb-33212aab7ba5","Type":"ContainerStarted","Data":"986adc966a1f428a26ef1fcc38a47dded87dad6c3c54e2b6d931b2171d229419"}
Nov 02 17:57:51.505494 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:51.505435    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podStartSLOduration=2.50541787 podStartE2EDuration="2.50541787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:49 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:51.501136707 +0000 UTC m=+1687.992199887" watchObservedRunningTime="2026-11-02 17:57:51.50541787 +0000 UTC m=+1687.996481026"
Nov 02 17:57:52.261399 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:52.261367    4456 patch_prober.go:29] interesting pod/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 17:57:52.261562 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:52.261421    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 17:57:52.265617 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:52.265592    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:52.271657 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:52.271633    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:52.452293 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:52.452261    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:52.453264 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:52.453240    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 17:57:52.606283 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:52.606194    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:52.807031 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:52.806958    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:52.964792 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:52.964759    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:53.461920 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.461891    4456 generic.go:356] "Generic (PLEG): container finished" podID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerID="54ec2269cb5bd8ac76efbd725adbaeb88ab8284252404474208d847effc04e1b" exitCode=0
Nov 02 17:57:53.462467 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.461957    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" event={"ID":"0a41f8f2-1497-476f-b391-8b24c9c7f7f4","Type":"ContainerDied","Data":"54ec2269cb5bd8ac76efbd725adbaeb88ab8284252404474208d847effc04e1b"}
Nov 02 17:57:53.462467 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.462010    4456 scope.go:122] "RemoveContainer" containerID="8a129e0cab5c942205679f6b457e66733b2f8b0cc96304573dab32c95c14d867"
Nov 02 17:57:53.817741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.817715    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:57:53.947871 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.927400    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-podinfo\") pod \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") "
Nov 02 17:57:53.948195 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.948167    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-credential-socket\") pod \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") "
Nov 02 17:57:53.948399 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.948378    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istiod-ca-cert\") pod \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") "
Nov 02 17:57:53.948547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.948517    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-credential-socket" pod "0a41f8f2-1497-476f-b391-8b24c9c7f7f4" (UID: "0a41f8f2-1497-476f-b391-8b24c9c7f7f4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:53.948710 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.948548    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-kube-api-access-t7kh4\" (UniqueName: \"kubernetes.io/projected/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-kube-api-access-t7kh4\") pod \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") "
Nov 02 17:57:53.948964 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.948943    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-envoy\") pod \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") "
Nov 02 17:57:53.949212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.949194    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-workload-socket\") pod \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") "
Nov 02 17:57:53.949340 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.949325    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-data\") pod \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") "
Nov 02 17:57:53.949469 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.949456    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-workload-certs\") pod \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") "
Nov 02 17:57:53.949592 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.949578    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-token\" (UniqueName: \"kubernetes.io/projected/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-token\") pod \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\" (UID: \"0a41f8f2-1497-476f-b391-8b24c9c7f7f4\") "
Nov 02 17:57:53.950462 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.950439    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:53.950895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.949227    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istiod-ca-cert" pod "0a41f8f2-1497-476f-b391-8b24c9c7f7f4" (UID: "0a41f8f2-1497-476f-b391-8b24c9c7f7f4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:57:53.951603 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.951205    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-data" pod "0a41f8f2-1497-476f-b391-8b24c9c7f7f4" (UID: "0a41f8f2-1497-476f-b391-8b24c9c7f7f4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:53.951741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.951372    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-workload-socket" pod "0a41f8f2-1497-476f-b391-8b24c9c7f7f4" (UID: "0a41f8f2-1497-476f-b391-8b24c9c7f7f4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:53.951910 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.951530    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-workload-certs" pod "0a41f8f2-1497-476f-b391-8b24c9c7f7f4" (UID: "0a41f8f2-1497-476f-b391-8b24c9c7f7f4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:53.952235 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.952210    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-podinfo" pod "0a41f8f2-1497-476f-b391-8b24c9c7f7f4" (UID: "0a41f8f2-1497-476f-b391-8b24c9c7f7f4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:57:53.954924 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.954881    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-kube-api-access-t7kh4" pod "0a41f8f2-1497-476f-b391-8b24c9c7f7f4" (UID: "0a41f8f2-1497-476f-b391-8b24c9c7f7f4"). InnerVolumeSpecName "kube-api-access-t7kh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:53.956098 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.956078    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-envoy" pod "0a41f8f2-1497-476f-b391-8b24c9c7f7f4" (UID: "0a41f8f2-1497-476f-b391-8b24c9c7f7f4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:57:53.956561 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:53.956541    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-token" pod "0a41f8f2-1497-476f-b391-8b24c9c7f7f4" (UID: "0a41f8f2-1497-476f-b391-8b24c9c7f7f4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:57:54.041651 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.041564    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:54.051919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.051886    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:54.051919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.051917    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:54.051919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.051932    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:54.052174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.051946    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-t7kh4\" (UniqueName: \"kubernetes.io/projected/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-kube-api-access-t7kh4\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:54.052174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.051959    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:54.052174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.051971    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:54.052174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.051981    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:54.052174 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.051992    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0a41f8f2-1497-476f-b391-8b24c9c7f7f4-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:57:54.197410 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.194107    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:54.472675 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.472647    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"
Nov 02 17:57:54.472675 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.472667    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk" event={"ID":"0a41f8f2-1497-476f-b391-8b24c9c7f7f4","Type":"ContainerDied","Data":"c6ff67e2a366daf78949aa694518cce2d007bd0eca1783c54f56c5ce551d474d"}
Nov 02 17:57:54.473228 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.472752    4456 scope.go:122] "RemoveContainer" containerID="54ec2269cb5bd8ac76efbd725adbaeb88ab8284252404474208d847effc04e1b"
Nov 02 17:57:54.545756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.545723    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"]
Nov 02 17:57:54.557419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.557375    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-api-b-227833-openshift-default-6784bcbf85-nxmlk"]
Nov 02 17:57:54.630392 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:54.630003    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:55.389080 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.389047    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"
Nov 02 17:57:55.389298 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.389121    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"
Nov 02 17:57:55.617928 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.617886    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" path="/var/lib/kubelet/pods/0a41f8f2-1497-476f-b391-8b24c9c7f7f4/volumes"
Nov 02 17:57:55.786961 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.786923    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:55.926441 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.926405    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"]
Nov 02 17:57:55.927410 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.927392    4456 cpu_manager.go:414] "containerMap: removing container" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy"
Nov 02 17:57:55.927491 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.927413    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy"
Nov 02 17:57:55.927491 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.927425    4456 cpu_manager.go:414] "containerMap: removing container" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy"
Nov 02 17:57:55.927491 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.927430    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy"
Nov 02 17:57:55.927637 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.927625    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy"
Nov 02 17:57:55.927692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.927639    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="0a41f8f2-1497-476f-b391-8b24c9c7f7f4" containerName="istio-proxy"
Nov 02 17:57:55.928483 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.928452    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:55.933560 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.933541    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"
Nov 02 17:57:55.949180 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.949151    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-872354-dockercfg-hw7xl\""
Nov 02 17:57:55.949329 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.949215    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-872354\""
Nov 02 17:57:55.956713 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:55.956683    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"]
Nov 02 17:57:56.017192 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.017139    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g\" (UID: \"44b66f41-99ae-446b-a50a-2ec0b71563a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"
Nov 02 17:57:56.019136 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.019078    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qbrdf\" (UniqueName: \"kubernetes.io/projected/44b66f41-99ae-446b-a50a-2ec0b71563a4-kube-api-access-qbrdf\") pod \"payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g\" (UID: \"44b66f41-99ae-446b-a50a-2ec0b71563a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"
Nov 02 17:57:56.125888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.123516    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g\" (UID: \"44b66f41-99ae-446b-a50a-2ec0b71563a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"
Nov 02 17:57:56.125888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.123631    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qbrdf\" (UniqueName: \"kubernetes.io/projected/44b66f41-99ae-446b-a50a-2ec0b71563a4-kube-api-access-qbrdf\") pod \"payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g\" (UID: \"44b66f41-99ae-446b-a50a-2ec0b71563a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"
Nov 02 17:57:56.125888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.125087    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g\" (UID: \"44b66f41-99ae-446b-a50a-2ec0b71563a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"
Nov 02 17:57:56.138841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.131896    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"]
Nov 02 17:57:56.296839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.296762    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 17:57:56.302156 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.302116    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qbrdf\" (UniqueName: \"kubernetes.io/projected/44b66f41-99ae-446b-a50a-2ec0b71563a4-kube-api-access-qbrdf\") pod \"payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g\" (UID: \"44b66f41-99ae-446b-a50a-2ec0b71563a4\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"
Nov 02 17:57:56.307749 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.307656    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"]
Nov 02 17:57:56.328276 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.328238    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb\" (UID: \"cf85b137-2b1e-4e22-8f6e-744a02ea4e46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 17:57:56.328504 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.328486    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9w876\" (UniqueName: \"kubernetes.io/projected/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-kube-api-access-9w876\") pod \"payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb\" (UID: \"cf85b137-2b1e-4e22-8f6e-744a02ea4e46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 17:57:56.430142 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.430060    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9w876\" (UniqueName: \"kubernetes.io/projected/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-kube-api-access-9w876\") pod \"payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb\" (UID: \"cf85b137-2b1e-4e22-8f6e-744a02ea4e46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 17:57:56.430296 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.430219    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb\" (UID: \"cf85b137-2b1e-4e22-8f6e-744a02ea4e46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 17:57:56.430988 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.430969    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume\") pod \"payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb\" (UID: \"cf85b137-2b1e-4e22-8f6e-744a02ea4e46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 17:57:56.454199 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.454168    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9w876\" (UniqueName: \"kubernetes.io/projected/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-kube-api-access-9w876\") pod \"payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb\" (UID: \"cf85b137-2b1e-4e22-8f6e-744a02ea4e46\") " pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 17:57:56.551047 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.551016    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"
Nov 02 17:57:56.559324 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.559294    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:57:56.644908 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.644145    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 17:57:56.925380 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:56.925351    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"]
Nov 02 17:57:56.925953 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:57:56.925922    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod44b66f41_99ae_446b_a50a_2ec0b71563a4.slice/crio-af7639a1c401b49f95ce04b0dc0df7c785bd5660b007f844f58d72990f9b9823 WatchSource:0}: Error finding container af7639a1c401b49f95ce04b0dc0df7c785bd5660b007f844f58d72990f9b9823: Status 404 returned error can't find the container with id af7639a1c401b49f95ce04b0dc0df7c785bd5660b007f844f58d72990f9b9823
Nov 02 17:57:57.012466 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:57:57.012438    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcf85b137_2b1e_4e22_8f6e_744a02ea4e46.slice/crio-1309254054a37595d00abffd63c04c21ea5ae37f9a2f91210c745674592bb4be WatchSource:0}: Error finding container 1309254054a37595d00abffd63c04c21ea5ae37f9a2f91210c745674592bb4be: Status 404 returned error can't find the container with id 1309254054a37595d00abffd63c04c21ea5ae37f9a2f91210c745674592bb4be
Nov 02 17:57:57.014524 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:57.014495    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"]
Nov 02 17:57:57.538338 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:57.538291    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" event={"ID":"44b66f41-99ae-446b-a50a-2ec0b71563a4","Type":"ContainerStarted","Data":"a15fc9b95bc579e3250f894c650e5c31574a16f7a9e567db93f2291e2e8e8b0a"}
Nov 02 17:57:57.538547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:57.538364    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" event={"ID":"44b66f41-99ae-446b-a50a-2ec0b71563a4","Type":"ContainerStarted","Data":"af7639a1c401b49f95ce04b0dc0df7c785bd5660b007f844f58d72990f9b9823"}
Nov 02 17:57:57.538547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:57.538397    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"
Nov 02 17:57:57.540048 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:57.540022    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" event={"ID":"cf85b137-2b1e-4e22-8f6e-744a02ea4e46","Type":"ContainerStarted","Data":"1f50936a9578f37077e2da181b984035ceacd77c575058ecec4a61436289a702"}
Nov 02 17:57:57.540191 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:57.540064    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" event={"ID":"cf85b137-2b1e-4e22-8f6e-744a02ea4e46","Type":"ContainerStarted","Data":"1309254054a37595d00abffd63c04c21ea5ae37f9a2f91210c745674592bb4be"}
Nov 02 17:57:57.540191 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:57.540143    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 17:57:57.581868 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:57.581785    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" podStartSLOduration=2.581768562 podStartE2EDuration="2.581768562s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:57.579568014 +0000 UTC m=+1694.070631187" watchObservedRunningTime="2026-11-02 17:57:57.581768562 +0000 UTC m=+1694.072831727"
Nov 02 17:57:57.624793 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:57.624745    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" podStartSLOduration=1.624730435 podStartE2EDuration="1.624730435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:57:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:57:57.620183086 +0000 UTC m=+1694.111246246" watchObservedRunningTime="2026-11-02 17:57:57.624730435 +0000 UTC m=+1694.115793592"
Nov 02 17:57:58.303234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:57:58.303194    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:00.124258 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:00.124219    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:00.324858 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:00.324802    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:00.460463 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:00.460429    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:01.952520 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:01.952485    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:03.733685 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:03.733644    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:05.850120 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:05.850081    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:07.753851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:07.751337    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:08.551452 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:08.551422    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"
Nov 02 17:58:08.551637 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:08.551513    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 17:58:09.743428 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:09.743380    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:11.686178 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:11.686141    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:12.045795 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:12.045702    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:13.552364 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:13.552323    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:15.257016 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:15.256983    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:15.414673 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:15.414642    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:16.167581 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:58:16.167533    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:58:17.001026 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:17.000959    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:18.852977 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:18.852940    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:20.556390 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:20.556358    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:22.148349 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:22.146098    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:23.751288 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:23.751253    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:25.323138 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:25.323099    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:26.914459 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:26.914426    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:28.422065 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:28.422032    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:28.564300 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:28.564259    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:28.702161 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:28.702123    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:30.246122 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:30.246083    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:31.217109 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.217071    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"]
Nov 02 17:58:31.224496 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.224471    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.242783 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.242756    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"]
Nov 02 17:58:31.248539 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.248512    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8a209f-openshift-default-dockercfg-whvdh\""
Nov 02 17:58:31.316280 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.316237    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-envoy\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.316482 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.316351    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-workload-certs\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.316482 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.316396    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-workload-socket\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.316627 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.316509    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-podinfo\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.316627 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.316551    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-token\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.316627 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.316591    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istiod-ca-cert\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.316796 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.316641    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mfs8x\" (UniqueName: \"kubernetes.io/projected/81fd82a8-2bbf-43c8-962c-f8384287dc1f-kube-api-access-mfs8x\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.316796 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.316762    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-credential-socket\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.316943 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.316797    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-data\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.418447 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.418400    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-credential-socket\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.418641 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.418467    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-data\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.418641 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.418521    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-envoy\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.418641 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.418630    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-workload-certs\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.418866 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.418670    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-workload-socket\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.418866 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.418833    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-podinfo\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.419002 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.418883    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-token\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.419002 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.418926    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istiod-ca-cert\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.419002 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.418961    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-data\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.419158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.419092    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-workload-socket\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.419158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.419093    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mfs8x\" (UniqueName: \"kubernetes.io/projected/81fd82a8-2bbf-43c8-962c-f8384287dc1f-kube-api-access-mfs8x\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.419547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.419528    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-workload-certs\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.419653 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.419633    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istiod-ca-cert\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.419730 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.419698    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-credential-socket\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.422079 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.422051    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-podinfo\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.422225 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.422171    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-envoy\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.443228 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.443199    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-token\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.444196 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.444172    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mfs8x\" (UniqueName: \"kubernetes.io/projected/81fd82a8-2bbf-43c8-962c-f8384287dc1f-kube-api-access-mfs8x\") pod \"e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") " pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.547508 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.547414    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:31.832605 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.832511    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:31.912031 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.908314    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"]
Nov 02 17:58:31.913554 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.913514    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:31.914379 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.913599    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:31.914379 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:31.913631    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:58:32.008567 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:32.008533    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" event={"ID":"81fd82a8-2bbf-43c8-962c-f8384287dc1f","Type":"ContainerStarted","Data":"730da757c8ffa0ec7438177532c8a6ce99b4b9d761e88b8e9e1bb45179e1eca6"}
Nov 02 17:58:32.008740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:32.008604    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" event={"ID":"81fd82a8-2bbf-43c8-962c-f8384287dc1f","Type":"ContainerStarted","Data":"b86b3ef9420d3f4216826fe71374370eb03bb3f93f859161af397d62b99ab672"}
Nov 02 17:58:32.057359 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:32.057301    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" podStartSLOduration=1.057286588 podStartE2EDuration="1.057286588s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:58:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:58:32.057183353 +0000 UTC m=+1728.548246528" watchObservedRunningTime="2026-11-02 17:58:32.057286588 +0000 UTC m=+1728.548349748"
Nov 02 17:58:32.551841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:32.548433    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:32.551841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:32.551279    4456 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 02 17:58:32.551841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:32.551344    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 02 17:58:33.525455 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:33.525416    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:33.566999 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:33.566956    4456 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 02 17:58:33.567431 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:33.567021    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 02 17:58:34.548649 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:34.548613    4456 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 02 17:58:34.548838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:34.548664    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 02 17:58:35.146968 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:35.146932    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:35.548172 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:35.548137    4456 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 02 17:58:35.548399 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:35.548199    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 02 17:58:36.121455 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:36.121382    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:36.269537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:36.269498    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:36.548543 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:36.548510    4456 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 02 17:58:36.548741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:36.548574    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 02 17:58:36.766497 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:36.766458    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:37.548106 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:37.548075    4456 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused" start-of-body=
Nov 02 17:58:37.548510 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:37.548136    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": dial tcp 10.134.0.191:15021: connect: connection refused"
Nov 02 17:58:38.304512 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:38.304477    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:38.553254 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:38.553224    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:38.553740 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:38.553594    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:38.554777 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:38.554715    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 17:58:39.290668 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:39.290627    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:39.431661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:39.431630    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:39.933854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:39.933793    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:41.701161 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:41.700933    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:41.853526 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:41.853466    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:42.138584 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:42.138497    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:43.675006 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:43.674962    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:43.820442 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:43.820401    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:44.916808 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:44.916778    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:45.713769 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:45.713732    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:46.169313 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:58:46.169224    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:58:46.495072 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:46.495041    4456 scope.go:122] "RemoveContainer" containerID="5241d39e91ac4d84aff35eac60f4c39c00b61c2b5bf77e9958329cf519f18cbc"
Nov 02 17:58:46.512231 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:46.512203    4456 scope.go:122] "RemoveContainer" containerID="49a65df5ba36041e13d2bbcbed7821d59f5a280bcda603271fc0845d2985dd6a"
Nov 02 17:58:46.527311 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:46.527284    4456 scope.go:122] "RemoveContainer" containerID="72949d142729cd47f40c5b13b9d137f93f6fb37ba572784be324dc2b0d21ebab"
Nov 02 17:58:47.558944 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:47.558907    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:48.671505 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:48.671470    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:48.811132 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:48.811088    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:48.950877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:48.950841    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"]
Nov 02 17:58:48.951147 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:48.951118    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh" podUID="426b74e4-82bb-46fc-b57f-f6ae93bcbefc" containerName="istio-proxy" containerID="cri-o://ac1d6511fe0a00fd60087a38695324eeed479fc719e25ab20d0cbc5b050d78e3" gracePeriod=30
Nov 02 17:58:49.358101 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:49.358024    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:50.992835 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:50.992291    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:51.963354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:51.963312    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:52.740093 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:52.740054    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:54.035972 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.035939    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"]
Nov 02 17:58:54.047942 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.047916    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6744769d-ts85l"]
Nov 02 17:58:54.264094 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.264056    4456 generic.go:356] "Generic (PLEG): container finished" podID="426b74e4-82bb-46fc-b57f-f6ae93bcbefc" containerID="ac1d6511fe0a00fd60087a38695324eeed479fc719e25ab20d0cbc5b050d78e3" exitCode=0
Nov 02 17:58:54.264094 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.264086    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh" event={"ID":"426b74e4-82bb-46fc-b57f-f6ae93bcbefc","Type":"ContainerDied","Data":"ac1d6511fe0a00fd60087a38695324eeed479fc719e25ab20d0cbc5b050d78e3"}
Nov 02 17:58:54.264290 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.264165    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh" event={"ID":"426b74e4-82bb-46fc-b57f-f6ae93bcbefc","Type":"ContainerDied","Data":"5865296a299bec7320dfdf5c8a7916d8c4a2a286029ce23c3b738fed503f57bd"}
Nov 02 17:58:54.264290 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.264183    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5865296a299bec7320dfdf5c8a7916d8c4a2a286029ce23c3b738fed503f57bd"
Nov 02 17:58:54.266427 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.266408    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:58:54.338596 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.338549    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-token\" (UniqueName: \"kubernetes.io/projected/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-token\") pod \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") "
Nov 02 17:58:54.338796 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.338648    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-workload-socket\") pod \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") "
Nov 02 17:58:54.338796 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.338724    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-data\") pod \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") "
Nov 02 17:58:54.338796 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.338760    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-kube-api-access-sd4jt\" (UniqueName: \"kubernetes.io/projected/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-kube-api-access-sd4jt\") pod \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") "
Nov 02 17:58:54.338796 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.338796    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-envoy\") pod \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") "
Nov 02 17:58:54.339210 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.339184    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istiod-ca-cert\") pod \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") "
Nov 02 17:58:54.340244 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.340224    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-workload-certs\") pod \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") "
Nov 02 17:58:54.340831 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.340792    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-credential-socket\") pod \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") "
Nov 02 17:58:54.341261 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.341244    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-podinfo\") pod \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\" (UID: \"426b74e4-82bb-46fc-b57f-f6ae93bcbefc\") "
Nov 02 17:58:54.341850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.339545    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-workload-socket" pod "426b74e4-82bb-46fc-b57f-f6ae93bcbefc" (UID: "426b74e4-82bb-46fc-b57f-f6ae93bcbefc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:54.341850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.339789    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-data" pod "426b74e4-82bb-46fc-b57f-f6ae93bcbefc" (UID: "426b74e4-82bb-46fc-b57f-f6ae93bcbefc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:54.341850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.340135    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istiod-ca-cert" pod "426b74e4-82bb-46fc-b57f-f6ae93bcbefc" (UID: "426b74e4-82bb-46fc-b57f-f6ae93bcbefc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 17:58:54.341850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.340695    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-workload-certs" pod "426b74e4-82bb-46fc-b57f-f6ae93bcbefc" (UID: "426b74e4-82bb-46fc-b57f-f6ae93bcbefc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:54.341850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.341148    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-credential-socket" pod "426b74e4-82bb-46fc-b57f-f6ae93bcbefc" (UID: "426b74e4-82bb-46fc-b57f-f6ae93bcbefc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:54.342715 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.342685    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:58:54.342887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.342719    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:58:54.342887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.342733    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:58:54.342887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.342747    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:58:54.342887 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.342760    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:58:54.344639 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.344601    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-kube-api-access-sd4jt" pod "426b74e4-82bb-46fc-b57f-f6ae93bcbefc" (UID: "426b74e4-82bb-46fc-b57f-f6ae93bcbefc"). InnerVolumeSpecName "kube-api-access-sd4jt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:54.344985 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.344960    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-token" pod "426b74e4-82bb-46fc-b57f-f6ae93bcbefc" (UID: "426b74e4-82bb-46fc-b57f-f6ae93bcbefc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 17:58:54.345238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.345209    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-envoy" pod "426b74e4-82bb-46fc-b57f-f6ae93bcbefc" (UID: "426b74e4-82bb-46fc-b57f-f6ae93bcbefc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 17:58:54.346316 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.346290    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-podinfo" pod "426b74e4-82bb-46fc-b57f-f6ae93bcbefc" (UID: "426b74e4-82bb-46fc-b57f-f6ae93bcbefc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 17:58:54.367762 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.367654    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:54.470245 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.470206    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sd4jt\" (UniqueName: \"kubernetes.io/projected/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-kube-api-access-sd4jt\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:58:54.470245 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.470246    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:58:54.470245 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.470259    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:58:54.470557 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:54.470273    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/426b74e4-82bb-46fc-b57f-f6ae93bcbefc-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 17:58:55.275854 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:55.275777    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"
Nov 02 17:58:55.357683 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:55.357645    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"]
Nov 02 17:58:55.369369 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:55.369339    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-t2vbh"]
Nov 02 17:58:55.620961 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:55.620837    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="426b74e4-82bb-46fc-b57f-f6ae93bcbefc" path="/var/lib/kubelet/pods/426b74e4-82bb-46fc-b57f-f6ae93bcbefc/volumes"
Nov 02 17:58:55.621504 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:55.621483    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="eb819cb7-0963-4c99-91c4-24ac65143c75" path="/var/lib/kubelet/pods/eb819cb7-0963-4c99-91c4-24ac65143c75/volumes"
Nov 02 17:58:56.020385 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:56.020336    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:56.160460 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:56.160425    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:57.762849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:57.762257    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:58:59.438838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:58:59.438776    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:01.035842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:01.034894    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:02.681845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:02.674655    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:04.241225 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:04.241181    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:05.892378 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:05.892289    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:06.718973 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:06.718918    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:59:06.719150 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:06.719014    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:07.534336 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:07.534291    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:07.675676 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:07.675646    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:09.158972 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:09.158889    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:10.322973 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:10.322928    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:59:10.323532 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:10.323006    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:10.917108 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:10.917066    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:12.558428 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:12.558383    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:12.784981 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:12.784942    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:59:12.785155 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:12.785003    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:14.129699 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:14.129598    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:15.828264 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:15.828227    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:16.167664 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:59:16.167587    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:59:17.496637 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:17.496593    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:17.919201 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:17.919106    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"]
Nov 02 17:59:17.920073 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:17.920050    4456 cpu_manager.go:414] "containerMap: removing container" podUID="426b74e4-82bb-46fc-b57f-f6ae93bcbefc" containerName="istio-proxy"
Nov 02 17:59:17.920073 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:17.920071    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="426b74e4-82bb-46fc-b57f-f6ae93bcbefc" containerName="istio-proxy"
Nov 02 17:59:17.920286 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:17.920250    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="426b74e4-82bb-46fc-b57f-f6ae93bcbefc" containerName="istio-proxy"
Nov 02 17:59:17.973609 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:17.973570    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"]
Nov 02 17:59:17.973796 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:17.973648    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.056212 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.056178    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408afe2d-75a0-4537-b58c-9ab31f161685-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.056406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.056259    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408afe2d-75a0-4537-b58c-9ab31f161685-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.056406 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.056339    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.056537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.056448    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.056599 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.056564    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hm9fl\" (UniqueName: \"kubernetes.io/projected/408afe2d-75a0-4537-b58c-9ab31f161685-kube-api-access-hm9fl\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.056662 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.056647    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.056772 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.056752    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408afe2d-75a0-4537-b58c-9ab31f161685-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.056919 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.056901    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.056984 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.056954    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.158459 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.158424    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.158459 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.158467    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.158716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.158491    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408afe2d-75a0-4537-b58c-9ab31f161685-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.158716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.158546    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408afe2d-75a0-4537-b58c-9ab31f161685-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.158716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.158568    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.158716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.158639    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.158716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.158707    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hm9fl\" (UniqueName: \"kubernetes.io/projected/408afe2d-75a0-4537-b58c-9ab31f161685-kube-api-access-hm9fl\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.159038 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.158765    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.159038 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.158858    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408afe2d-75a0-4537-b58c-9ab31f161685-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.159038 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.158917    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.159222 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.159111    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.159294 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.159274    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408afe2d-75a0-4537-b58c-9ab31f161685-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.159554 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.159413    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.159858 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.159673    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.161177 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.161149    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.161732 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.161709    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408afe2d-75a0-4537-b58c-9ab31f161685-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.182692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.182623    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408afe2d-75a0-4537-b58c-9ab31f161685-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.182692 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.182625    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hm9fl\" (UniqueName: \"kubernetes.io/projected/408afe2d-75a0-4537-b58c-9ab31f161685-kube-api-access-hm9fl\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.380283 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.380250    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:18.761989 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.761952    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"]
Nov 02 17:59:18.763488 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:59:18.763456    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod408afe2d_75a0_4537_b58c_9ab31f161685.slice/crio-cb1df1cbaf8bfb6078cbda186b01ed7e40095d1b0ba50d11718114d10c543589 WatchSource:0}: Error finding container cb1df1cbaf8bfb6078cbda186b01ed7e40095d1b0ba50d11718114d10c543589: Status 404 returned error can't find the container with id cb1df1cbaf8bfb6078cbda186b01ed7e40095d1b0ba50d11718114d10c543589
Nov 02 17:59:18.766443 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.766412    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:18.766522 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.766477    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:18.766522 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.766509    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:18.942198 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:18.942162    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" event={"ID":"408afe2d-75a0-4537-b58c-9ab31f161685","Type":"ContainerStarted","Data":"cb1df1cbaf8bfb6078cbda186b01ed7e40095d1b0ba50d11718114d10c543589"}
Nov 02 17:59:19.506880 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:19.506843    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:19.953354 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:19.953308    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" event={"ID":"408afe2d-75a0-4537-b58c-9ab31f161685","Type":"ContainerStarted","Data":"19ff9701b232887fc6c8cb51d7c49ea1498230cb1843842011b53e7a4b6f7336"}
Nov 02 17:59:20.033062 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:20.032470    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" podStartSLOduration=3.032446994 podStartE2EDuration="3.032446994s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:17 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:20.025265616 +0000 UTC m=+1776.516328774" watchObservedRunningTime="2026-11-02 17:59:20.032446994 +0000 UTC m=+1776.523510156"
Nov 02 17:59:20.391893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:20.386979    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:20.391893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:20.387116    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 17:59:20.391893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:20.387172    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 17:59:20.467583 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:20.467547    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:20.924187 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:20.924083    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"]
Nov 02 17:59:21.028024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.027376    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:21.138193 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.134115    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"]
Nov 02 17:59:21.138193 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.134266    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.172893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.170058    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5fd3e0-openshift-default-dockercfg-kxlxh\""
Nov 02 17:59:21.194088 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.190626    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:21.243373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.241190    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999df778-cd9b-4b41-852d-4661a7326483-istiod-ca-cert\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.243373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.241268    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999df778-cd9b-4b41-852d-4661a7326483-istio-token\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.243373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.241358    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999df778-cd9b-4b41-852d-4661a7326483-istio-podinfo\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.243373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.241388    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-credential-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.243373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.241491    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-istio-envoy\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.243373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.241644    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-istio-data\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.243373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.241677    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-workload-certs\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.243373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.241714    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-workload-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.243373 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.241743    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q2gdq\" (UniqueName: \"kubernetes.io/projected/999df778-cd9b-4b41-852d-4661a7326483-kube-api-access-q2gdq\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.361843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.345051    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999df778-cd9b-4b41-852d-4661a7326483-istiod-ca-cert\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.361843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.345125    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999df778-cd9b-4b41-852d-4661a7326483-istio-token\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.361843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.345216    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999df778-cd9b-4b41-852d-4661a7326483-istio-podinfo\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.361843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.345247    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-credential-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.361843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.345349    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-istio-envoy\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.361843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.345502    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-istio-data\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.361843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.345538    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-workload-certs\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.361843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.345578    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-workload-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.361843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.345603    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q2gdq\" (UniqueName: \"kubernetes.io/projected/999df778-cd9b-4b41-852d-4661a7326483-kube-api-access-q2gdq\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.361843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.346922    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999df778-cd9b-4b41-852d-4661a7326483-istiod-ca-cert\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.361843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.347882    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-credential-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.361843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.348421    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-workload-certs\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.361843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.348673    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-workload-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.361843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.349695    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-istio-data\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.413889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.368458    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-istio-envoy\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.413889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.384288    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 17:59:21.413889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.384341    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 17:59:21.413889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.386355    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999df778-cd9b-4b41-852d-4661a7326483-istio-podinfo\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.413889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.400165    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999df778-cd9b-4b41-852d-4661a7326483-istio-token\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.436985 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.436364    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q2gdq\" (UniqueName: \"kubernetes.io/projected/999df778-cd9b-4b41-852d-4661a7326483-kube-api-access-q2gdq\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.660909 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.660875    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:21.720886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.719946    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:59:21.720886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:21.719994    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:22.222275 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.222227    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"]
Nov 02 17:59:22.263104 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.263068    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"]
Nov 02 17:59:22.263342 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.263170    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.370859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.368705    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.370859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.368773    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0996b2fa-40e8-469c-9ace-75458691fa25-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.370859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.368907    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0996b2fa-40e8-469c-9ace-75458691fa25-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.370859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.369014    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.370859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.369046    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cx2dz\" (UniqueName: \"kubernetes.io/projected/0996b2fa-40e8-469c-9ace-75458691fa25-kube-api-access-cx2dz\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.370859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.369123    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0996b2fa-40e8-469c-9ace-75458691fa25-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.370859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.369158    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.370859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.369181    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.370859 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.369225    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.413898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.407215    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 17:59:22.413898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.407273    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 17:59:22.470697 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.470661    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0996b2fa-40e8-469c-9ace-75458691fa25-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.470934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.470716    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.470934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.470743    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.470934 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.470787    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.471157 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.471078    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.471157 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.471124    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0996b2fa-40e8-469c-9ace-75458691fa25-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.471284 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.471181    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0996b2fa-40e8-469c-9ace-75458691fa25-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.471284 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.471187    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.471400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.471292    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.471400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.471326    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cx2dz\" (UniqueName: \"kubernetes.io/projected/0996b2fa-40e8-469c-9ace-75458691fa25-kube-api-access-cx2dz\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.471525 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.471492    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.471839 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.471676    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.472099 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.472073    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.472220 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.472150    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0996b2fa-40e8-469c-9ace-75458691fa25-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.499415 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.497064    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"]
Nov 02 17:59:22.502842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.502221    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cx2dz\" (UniqueName: \"kubernetes.io/projected/0996b2fa-40e8-469c-9ace-75458691fa25-kube-api-access-cx2dz\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.502842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.502677    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0996b2fa-40e8-469c-9ace-75458691fa25-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.514964 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.514206    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.517215 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.517170    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0996b2fa-40e8-469c-9ace-75458691fa25-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:22.522070 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:59:22.522043    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod999df778_cd9b_4b41_852d_4661a7326483.slice/crio-e8ac58feb06de46a148ffed2ba94bf93173c751ea50dba769ff13cd0fe67b9fb WatchSource:0}: Error finding container e8ac58feb06de46a148ffed2ba94bf93173c751ea50dba769ff13cd0fe67b9fb: Status 404 returned error can't find the container with id e8ac58feb06de46a148ffed2ba94bf93173c751ea50dba769ff13cd0fe67b9fb
Nov 02 17:59:22.525964 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.525915    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:22.527944 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.526023    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:22.527944 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.526075    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:22.635037 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:22.635009    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:23.037866 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:23.034946    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" event={"ID":"999df778-cd9b-4b41-852d-4661a7326483","Type":"ContainerStarted","Data":"1f92044f46d783f762d4c677bea78e3b2f2e6ab30bddaea987c9b15454088386"}
Nov 02 17:59:23.037866 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:23.035017    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" event={"ID":"999df778-cd9b-4b41-852d-4661a7326483","Type":"ContainerStarted","Data":"e8ac58feb06de46a148ffed2ba94bf93173c751ea50dba769ff13cd0fe67b9fb"}
Nov 02 17:59:23.128050 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:23.123165    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podStartSLOduration=3.123146489 podStartE2EDuration="3.123146489s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:23.122766143 +0000 UTC m=+1779.613829317" watchObservedRunningTime="2026-11-02 17:59:23.123146489 +0000 UTC m=+1779.614209651"
Nov 02 17:59:23.144054 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:23.144018    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"]
Nov 02 17:59:23.161877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:23.160529    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:23.161877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:23.160621    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:23.161877 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:23.160664    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:23.176440 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:23.176339    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:23.327462 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:23.327394    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:23.387932 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:23.385234    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 17:59:23.387932 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:23.385294    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 17:59:23.697896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:23.681413    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:23.697896 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:23.681534    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:24.050017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.049875    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" event={"ID":"0996b2fa-40e8-469c-9ace-75458691fa25","Type":"ContainerStarted","Data":"0f61b587f093bc2a10aa5b15cd1853e4adc293fe1098512029b4141d26f10d42"}
Nov 02 17:59:24.050017 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.049934    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" event={"ID":"0996b2fa-40e8-469c-9ace-75458691fa25","Type":"ContainerStarted","Data":"bdd3bb397d2d235caac6e3e217c612baf7d6db4d168a86f42bc1ab293dc8256d"}
Nov 02 17:59:24.050298 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.050094    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:24.051879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.051529    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 17:59:24.164881 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.164754    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podStartSLOduration=2.164737769 podStartE2EDuration="2.164737769s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:22 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:24.150037371 +0000 UTC m=+1780.641100569" watchObservedRunningTime="2026-11-02 17:59:24.164737769 +0000 UTC m=+1780.655800929"
Nov 02 17:59:24.398911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.396042    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 17:59:24.398911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.396112    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 17:59:24.651321 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.651230    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:24.655084 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.655050    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:59:24.655232 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.655110    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 17:59:24.666362 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.666331    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"]
Nov 02 17:59:24.679464 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.678432    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.778778 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.778729    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.779008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.778789    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.779008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.778902    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.779008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.778927    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.779008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.778957    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.779274 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.779157    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.779274 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.779239    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.779400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.779356    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.779400 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.779389    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ccvc9\" (UniqueName: \"kubernetes.io/projected/5bb882f6-fd8f-46ae-9396-b52498c5d9db-kube-api-access-ccvc9\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.780106 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.779609    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"]
Nov 02 17:59:24.868647 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.868322    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"]
Nov 02 17:59:24.868925 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.868908    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"]
Nov 02 17:59:24.869044 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.868541    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:24.956879 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.940576    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.958616 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.947258    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.958796 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.958317    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ccvc9\" (UniqueName: \"kubernetes.io/projected/5bb882f6-fd8f-46ae-9396-b52498c5d9db-kube-api-access-ccvc9\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.959030 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.959013    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.959229 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.959210    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.960719 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.959375    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.960719 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.959576    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.960719 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.959988    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.960719 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.960019    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.960719 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.960056    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.960719 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.960289    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.960719 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.960392    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.961238 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.960993    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:24.961576 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:24.961552    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:25.027047 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.022591    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:25.046636 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.041663    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:25.080738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.079651    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.080738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.080061    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.080738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.080111    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.080738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.080149    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.080738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.080186    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gzxpq\" (UniqueName: \"kubernetes.io/projected/73849b3d-673e-41a8-aff1-db9f0a9cce85-kube-api-access-gzxpq\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.080738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.080302    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.080738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.080385    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73849b3d-673e-41a8-aff1-db9f0a9cce85-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.080738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.080421    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.080738 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.080459    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.099076 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.093218    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ccvc9\" (UniqueName: \"kubernetes.io/projected/5bb882f6-fd8f-46ae-9396-b52498c5d9db-kube-api-access-ccvc9\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:25.099076 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.093654    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:25.226891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.203221    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.226891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.203286    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.226891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.203324    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.226891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.203363    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gzxpq\" (UniqueName: \"kubernetes.io/projected/73849b3d-673e-41a8-aff1-db9f0a9cce85-kube-api-access-gzxpq\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.226891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.203550    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.226891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.203637    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73849b3d-673e-41a8-aff1-db9f0a9cce85-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.226891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.203679    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.226891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.203715    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.226891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.203786    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.226891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.205606    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-workload-certs\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.227790 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.227758    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-data\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.244567 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.231388    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-credential-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.254008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.240171    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-workload-socket\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.254008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.241188    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73849b3d-673e-41a8-aff1-db9f0a9cce85-istiod-ca-cert\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.254447 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.250838    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-podinfo\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.256671 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.256051    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-envoy\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.258423 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.257993    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-token\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.260756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.260735    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gzxpq\" (UniqueName: \"kubernetes.io/projected/73849b3d-673e-41a8-aff1-db9f0a9cce85-kube-api-access-gzxpq\") pod \"e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") " pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.372786 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.372540    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:25.450194 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.401795    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:25.450194 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.402139    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused" start-of-body=
Nov 02 17:59:25.450194 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.402178    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": dial tcp 10.134.0.192:15021: connect: connection refused"
Nov 02 17:59:25.450194 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.450009    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:59:25.450194 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.450092    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:25.574051 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.573897    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:25.637852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.635869    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:59:25.637852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:25.635953    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 17:59:26.020660 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:26.020558    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:26.201893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:26.197667    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:26.275494 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:26.274284    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"]
Nov 02 17:59:26.326419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:26.326354    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"]
Nov 02 17:59:26.327765 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:26.327726    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:26.327971 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:26.327805    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:26.327971 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:26.327868    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:26.333931 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:26.333900    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:26.334027 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:26.333977    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:26.334027 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:26.334021    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:26.405886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:26.404642    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:26.409922 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:26.406397    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:26.410743 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:26.410700    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 17:59:26.636385 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:26.636071    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:59:26.636385 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:26.636115    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 17:59:27.213234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:27.213192    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" event={"ID":"73849b3d-673e-41a8-aff1-db9f0a9cce85","Type":"ContainerStarted","Data":"321f32c6b20dd4c469c4740acf6ac58556d8ab5567e481fbe7905842a155c1f0"}
Nov 02 17:59:27.213502 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:27.213484    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" event={"ID":"73849b3d-673e-41a8-aff1-db9f0a9cce85","Type":"ContainerStarted","Data":"d1f60cf1a33f8ea16b7853b8f00764dcc02533e84f1f4e0ed57c2eb6b4b3585a"}
Nov 02 17:59:27.223323 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:27.223285    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" event={"ID":"5bb882f6-fd8f-46ae-9396-b52498c5d9db","Type":"ContainerStarted","Data":"8fd11b5da302643890d03cf8b90f90991b0ace49793a57bde1469d40e28383be"}
Nov 02 17:59:27.238633 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:27.238600    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" event={"ID":"5bb882f6-fd8f-46ae-9396-b52498c5d9db","Type":"ContainerStarted","Data":"141a56e13647c456e1524d4bceb0ae64ac8bb6c75d433cd9258439496b0d9621"}
Nov 02 17:59:27.635922 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:27.635832    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:59:27.635922 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:27.635904    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 17:59:27.799843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:27.785236    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:59:27.799843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:27.785298    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:27.926888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:27.926518    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:28.374176 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:28.372928    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:28.374867 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:28.374587    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 02 17:59:28.374867 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:28.374656    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 02 17:59:28.420716 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:28.420616    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:28.421158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:28.421040    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 02 17:59:28.421158 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:28.421093    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 02 17:59:28.429846 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:28.426395    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" podStartSLOduration=4.42637622 podStartE2EDuration="4.42637622s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:28.40041137 +0000 UTC m=+1784.891474545" watchObservedRunningTime="2026-11-02 17:59:28.42637622 +0000 UTC m=+1784.917439382"
Nov 02 17:59:28.488847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:28.483662    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podStartSLOduration=4.483641362 podStartE2EDuration="4.483641362s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:24 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:28.483168069 +0000 UTC m=+1784.974231268" watchObservedRunningTime="2026-11-02 17:59:28.483641362 +0000 UTC m=+1784.974704524"
Nov 02 17:59:28.619841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:28.594943    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:28.657912 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:28.654558    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:59:28.657912 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:28.654618    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 17:59:28.833847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:28.833276    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:29.030841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:29.016307    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:29.651523 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:29.648919    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:59:29.651523 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:29.648979    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 17:59:29.684399 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:29.684364    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 02 17:59:29.684706 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:29.684656    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 02 17:59:29.688438 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:29.687910    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 02 17:59:29.688438 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:29.688269    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 02 17:59:29.744061 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:29.732979    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:29.903380 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:29.902155    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:29.936756 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:29.933958    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:29.993477 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:29.993351    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:30.381886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:30.381727    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 02 17:59:30.381886 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:30.381790    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 02 17:59:30.410927 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:30.407120    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 02 17:59:30.410927 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:30.407175    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 02 17:59:30.507628 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:30.506911    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:30.551598 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:30.551325    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:30.589870 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:30.588897    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:30.639317 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:30.638767    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused" start-of-body=
Nov 02 17:59:30.639317 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:30.638847    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": dial tcp 10.134.0.194:15021: connect: connection refused"
Nov 02 17:59:30.765748 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:30.765333    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:30.765748 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:30.765664    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:59:30.870302 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:30.864754    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:59:31.379507 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:31.378935    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 02 17:59:31.379507 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:31.378998    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 02 17:59:31.436838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:31.435883    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 02 17:59:31.436838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:31.435947    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 02 17:59:31.660918 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:31.659334    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:31.660918 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:31.660859    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:31.661573 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:31.661557    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 17:59:31.983171 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:31.979481    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:32.373024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:32.372922    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 02 17:59:32.373024 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:32.372991    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 02 17:59:32.435844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:32.412574    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 02 17:59:32.435844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:32.412644    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 02 17:59:33.061843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.050879    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"]
Nov 02 17:59:33.249746 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.245164    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.250094 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.250035    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"]
Nov 02 17:59:33.360853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.360200    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.360853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.360345    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.360853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.360374    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778b9a36-6a7b-4251-a91a-999e358d8c5c-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.360853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.360403    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.360853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.360471    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.360853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.360506    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wz4ld\" (UniqueName: \"kubernetes.io/projected/778b9a36-6a7b-4251-a91a-999e358d8c5c-kube-api-access-wz4ld\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.360853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.360635    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.360853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.360668    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.360853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.360791    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.373114 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.373078    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused" start-of-body=
Nov 02 17:59:33.373263 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.373152    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": dial tcp 10.134.0.195:15021: connect: connection refused"
Nov 02 17:59:33.445657 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.415707    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 02 17:59:33.445657 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.415768    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 02 17:59:33.500621 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.500575    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.500845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.500638    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.500845 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.500727    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.500983 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.500847    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.501065 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.501030    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.501123 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.501058    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778b9a36-6a7b-4251-a91a-999e358d8c5c-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.501123 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.501090    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.501231 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.501178    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.501231 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.501218    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wz4ld\" (UniqueName: \"kubernetes.io/projected/778b9a36-6a7b-4251-a91a-999e358d8c5c-kube-api-access-wz4ld\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.519741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.502028    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.519741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.502549    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.519741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.503447    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.519741 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.503744    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.557849 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.542744    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778b9a36-6a7b-4251-a91a-999e358d8c5c-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.670893 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.670356    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.684852 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.683063    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.753869 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.723305    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:33.876841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.867583    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wz4ld\" (UniqueName: \"kubernetes.io/projected/778b9a36-6a7b-4251-a91a-999e358d8c5c-kube-api-access-wz4ld\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:34.126420 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:33.952516    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:34.158924 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:34.156654    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:34.417895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:34.414238    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused" start-of-body=
Nov 02 17:59:34.417895 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:34.414298    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": dial tcp 10.134.0.196:15021: connect: connection refused"
Nov 02 17:59:34.580865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:34.580730    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:34.587892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:34.581919    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:34.587892 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:34.585660    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 17:59:35.408809 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:35.408778    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:35.409278 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:35.409245    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:35.410547 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:35.410526    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 17:59:35.830064 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:35.830022    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:36.191148 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:36.191114    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:36.348925 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:36.348882    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:36.718008 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:36.717967    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:59:36.718189 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:36.718053    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:37.010994 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.010899    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:37.256287 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.256255    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"]
Nov 02 17:59:37.477768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.476789    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"]
Nov 02 17:59:37.477768 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.476993    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.508542 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.507973    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:37.548234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.547912    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.548234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.547969    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z2dvq\" (UniqueName: \"kubernetes.io/projected/551602c4-254b-4532-8961-a98bf0f81089-kube-api-access-z2dvq\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.548234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.548043    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.548234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.548110    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551602c4-254b-4532-8961-a98bf0f81089-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.548234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.548155    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.548234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.548194    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551602c4-254b-4532-8961-a98bf0f81089-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.548234 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.548233    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551602c4-254b-4532-8961-a98bf0f81089-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.553663 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.548276    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.553663 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.548314    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.650145 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.650106    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.650358 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.650242    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.650358 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.650277    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z2dvq\" (UniqueName: \"kubernetes.io/projected/551602c4-254b-4532-8961-a98bf0f81089-kube-api-access-z2dvq\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.650358 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.650310    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.650572 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.650375    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551602c4-254b-4532-8961-a98bf0f81089-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.650572 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.650419    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.650572 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.650459    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551602c4-254b-4532-8961-a98bf0f81089-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.650572 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.650498    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551602c4-254b-4532-8961-a98bf0f81089-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.650572 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.650541    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.651191 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.651163    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-workload-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.651347 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.651260    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-credential-socket\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.651458 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.651278    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-istio-data\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.651561 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.651514    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-workload-certs\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.652055 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.652022    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551602c4-254b-4532-8961-a98bf0f81089-istiod-ca-cert\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.654832 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.654798    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-istio-envoy\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.654991 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.654950    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551602c4-254b-4532-8961-a98bf0f81089-istio-podinfo\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.681278 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.681221    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551602c4-254b-4532-8961-a98bf0f81089-istio-token\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.681785 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.681675    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z2dvq\" (UniqueName: \"kubernetes.io/projected/551602c4-254b-4532-8961-a98bf0f81089-kube-api-access-z2dvq\") pod \"e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") " pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.699481 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.699443    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:37.746164 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.745834    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:37.887239 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.887199    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"]
Nov 02 17:59:37.891892 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:59:37.891796    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod778b9a36_6a7b_4251_a91a_999e358d8c5c.slice/crio-d903a26c6171a2534f2d5af1688376f441f68b47cb471efc27be8ad241043786 WatchSource:0}: Error finding container d903a26c6171a2534f2d5af1688376f441f68b47cb471efc27be8ad241043786: Status 404 returned error can't find the container with id d903a26c6171a2534f2d5af1688376f441f68b47cb471efc27be8ad241043786
Nov 02 17:59:37.900490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.899396    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:37.900490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.899498    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:37.900490 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:37.899541    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:38.083841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.077459    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" event={"ID":"778b9a36-6a7b-4251-a91a-999e358d8c5c","Type":"ContainerStarted","Data":"d903a26c6171a2534f2d5af1688376f441f68b47cb471efc27be8ad241043786"}
Nov 02 17:59:38.200955 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.200873    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"]
Nov 02 17:59:38.258891 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.256369    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:38.358177 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.358138    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"]
Nov 02 17:59:38.358558 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.358541    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.416840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.416762    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fbbbf8-openshift-default-dockercfg-b9pj2\""
Nov 02 17:59:38.511470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.508484    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-b75mv\" (UniqueName: \"kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-kube-api-access-b75mv\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.511470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.508944    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.511470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.509052    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.511470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.509105    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.511470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.509175    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd34da0-9b90-458d-9581-ed9caf3daa11-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.511470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.509376    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.511470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.509451    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.511470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.509522    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.511470 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.509554    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.621451 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.620609    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.621451 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.620711    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.621451 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.620782    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.621451 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.620837    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.621451 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.620981    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-b75mv\" (UniqueName: \"kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-kube-api-access-b75mv\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.621451 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.621024    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.621451 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.621107    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.621451 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.621158    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.621451 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.621263    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd34da0-9b90-458d-9581-ed9caf3daa11-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.730521 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.697925    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"]
Nov 02 17:59:38.730521 ip-10-0-141-87 kubenswrapper[4456]: W1102 17:59:38.699406    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod551602c4_254b_4532_8961_a98bf0f81089.slice/crio-39a7fb1949a9a5d2645eaa5f04aa88de3c666d66cd68aace8fb45d834bbc78d3 WatchSource:0}: Error finding container 39a7fb1949a9a5d2645eaa5f04aa88de3c666d66cd68aace8fb45d834bbc78d3: Status 404 returned error can't find the container with id 39a7fb1949a9a5d2645eaa5f04aa88de3c666d66cd68aace8fb45d834bbc78d3
Nov 02 17:59:38.730521 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.703262    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:38.730521 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.703358    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:38.730521 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.703400    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:38.948002 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.947752    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd34da0-9b90-458d-9581-ed9caf3daa11-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.948002 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.947888    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.948002 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.947965    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.948002 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.947980    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.948477 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.948042    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.950012 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.949983    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.950495 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.950470    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.952991 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.952922    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:38.953107 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:38.953036    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-b75mv\" (UniqueName: \"kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-kube-api-access-b75mv\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:39.084491 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:39.084443    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:39.123888 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:39.119352    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" event={"ID":"778b9a36-6a7b-4251-a91a-999e358d8c5c","Type":"ContainerStarted","Data":"b40a2aae2b6ef5d220e6c8a56acd93bfa30d09d873cc836d366f512f7dc799f8"}
Nov 02 17:59:39.130437 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:39.130331    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" event={"ID":"551602c4-254b-4532-8961-a98bf0f81089","Type":"ContainerStarted","Data":"39a7fb1949a9a5d2645eaa5f04aa88de3c666d66cd68aace8fb45d834bbc78d3"}
Nov 02 17:59:39.142513 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:39.142462    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:39.158382 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:39.158351    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:39.160505 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:39.160474    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 02 17:59:39.160685 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:39.160559    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 02 17:59:39.202838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:39.195126    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podStartSLOduration=7.195105926 podStartE2EDuration="7.195105926s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:39.177873732 +0000 UTC m=+1795.668936919" watchObservedRunningTime="2026-11-02 17:59:39.195105926 +0000 UTC m=+1795.686169088"
Nov 02 17:59:39.683690 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:39.681806    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:40.142012 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:40.141893    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" event={"ID":"551602c4-254b-4532-8961-a98bf0f81089","Type":"ContainerStarted","Data":"6a3d5cd7a3cf30d356953f12d067b1b473d9827087fa6525e30fb999560de281"}
Nov 02 17:59:40.158670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:40.158639    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 02 17:59:40.158825 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:40.158687    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 02 17:59:40.210690 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:40.210630    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" podStartSLOduration=3.210614182 podStartE2EDuration="3.210614182s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:37 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:40.205840144 +0000 UTC m=+1796.696903317" watchObservedRunningTime="2026-11-02 17:59:40.210614182 +0000 UTC m=+1796.701677417"
Nov 02 17:59:40.321007 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:40.320946    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 17:59:40.321206 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:40.321027    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:40.746431 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:40.746391    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:40.748522 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:40.748491    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 02 17:59:40.748667 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:40.748540    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 02 17:59:41.158509 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:41.158470    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 02 17:59:41.158961 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:41.158523    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 02 17:59:41.747842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:41.714502    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:41.750388 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:41.748995    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 17:59:41.750971 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:41.750845    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 17:59:41.752431 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:41.749141    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 02 17:59:41.752604 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:41.752577    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 02 17:59:42.158498 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:42.158063    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 02 17:59:42.158498 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:42.158123    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 02 17:59:42.175052 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:42.175018    4456 generic.go:356] "Generic (PLEG): container finished" podID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerID="9d4c1c0786ab2e7395cf39120539ef4b85c634872ec650a8b5de0096e6ff0bd0" exitCode=0
Nov 02 17:59:42.176401 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:42.176379    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" event={"ID":"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d","Type":"ContainerDied","Data":"9d4c1c0786ab2e7395cf39120539ef4b85c634872ec650a8b5de0096e6ff0bd0"}
Nov 02 17:59:42.177379 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:42.177359    4456 scope.go:122] "RemoveContainer" containerID="9d4c1c0786ab2e7395cf39120539ef4b85c634872ec650a8b5de0096e6ff0bd0"
Nov 02 17:59:42.211842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:42.205145    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:42.211842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:42.205260    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:42.211842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:42.205350    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:42.471899 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:42.471791    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:42.472046 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:42.471926    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:42.472046 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:42.471977    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:42.478394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:42.478365    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"]
Nov 02 17:59:42.753726 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:42.753292    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 02 17:59:42.753726 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:42.753362    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 02 17:59:43.174477 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:43.159047    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 02 17:59:43.174477 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:43.159123    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 02 17:59:43.277328 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:43.277293    4456 generic.go:356] "Generic (PLEG): container finished" podID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerID="026fbcc527da33cce56513069ee46f158e36543f79011d0d0036bc53fe44819c" exitCode=0
Nov 02 17:59:43.277521 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:43.277348    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" event={"ID":"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21","Type":"ContainerDied","Data":"026fbcc527da33cce56513069ee46f158e36543f79011d0d0036bc53fe44819c"}
Nov 02 17:59:43.278433 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:43.277975    4456 scope.go:122] "RemoveContainer" containerID="026fbcc527da33cce56513069ee46f158e36543f79011d0d0036bc53fe44819c"
Nov 02 17:59:43.335844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:43.332010    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" event={"ID":"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d","Type":"ContainerStarted","Data":"33774b00458e580d97ee67142da150ad8d47b01d7485185ee8fef16f91745dda"}
Nov 02 17:59:43.377057 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:59:43.377018    4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@19-33767-0.service\": RecentStats: unable to find data in memory cache]"
Nov 02 17:59:43.377297 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:43.377274    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" event={"ID":"dbd34da0-9b90-458d-9581-ed9caf3daa11","Type":"ContainerStarted","Data":"3e83f28511fc73768824a8bafbe547677dbeba837ccadd9f8b6eeb2e836fa5e8"}
Nov 02 17:59:43.379090 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:43.379050    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:43.379230 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:43.379147    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:43.379230 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:43.379196    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:43.649482 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:43.639699    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:43.793050 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:43.781961    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 02 17:59:43.793050 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:43.782030    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 02 17:59:43.951419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:43.910584    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 17:59:43.951419 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:43.910652    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 17:59:44.177273 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:44.176937    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 02 17:59:44.177273 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:44.176988    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 02 17:59:44.209805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:44.202665    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 02 17:59:44.209805 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:44.202725    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 02 17:59:44.272379 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:44.272344    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:59:44.272537 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:44.272408    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:44.353963 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:44.322876    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podStartSLOduration=6.322854599 podStartE2EDuration="6.322854599s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:43.594319906 +0000 UTC m=+1800.085383087" watchObservedRunningTime="2026-11-02 17:59:44.322854599 +0000 UTC m=+1800.813917762"
Nov 02 17:59:44.460669 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:44.460625    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" event={"ID":"dbd34da0-9b90-458d-9581-ed9caf3daa11","Type":"ContainerStarted","Data":"a602b0afbc3a3601e8c878664a9fa511e8e88746ceb9ca1f1a2a019ef2cff058"}
Nov 02 17:59:44.464661 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:44.464627    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" event={"ID":"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21","Type":"ContainerStarted","Data":"092d682533ef4dd4fce564dfead035598355d2c0e076d963e5b9852be346f785"}
Nov 02 17:59:44.616296 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:44.615988    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 17:59:44.616296 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:44.616047    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 17:59:44.712195 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:44.711909    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:59:44.770642 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:44.769542    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 02 17:59:44.770642 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:44.769604    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 02 17:59:45.253915 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:45.253802    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 02 17:59:45.253915 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:45.253894    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 02 17:59:45.413947 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:45.323257    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:45.420853 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:45.420582    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:45.626385 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:45.624592    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 17:59:45.626385 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:45.624658    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 17:59:45.652666 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:45.652332    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:45.793865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:45.790995    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 02 17:59:45.793865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:45.791108    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 02 17:59:45.793865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:45.793794    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 02 17:59:45.793865 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:45.793862    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 02 17:59:45.797133 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:45.797095    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 17:59:45.888484 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:45.886374    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 17:59:46.058033 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.057989    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:46.151476 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.150989    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 17:59:46.158524 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.158210    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused" start-of-body=
Nov 02 17:59:46.162838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.158759    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": dial tcp 10.134.0.197:15021: connect: connection refused"
Nov 02 17:59:46.365856 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.356346    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:46.536842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.511934    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:46.619841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.608002    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 17:59:46.619841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.608059    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 17:59:46.641838 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:59:46.635646    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 17:59:46.641838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.636151    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:46.757843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.737968    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 02 17:59:46.757843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.738028    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 02 17:59:46.757843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.749585    4456 scope.go:122] "RemoveContainer" containerID="ac1d6511fe0a00fd60087a38695324eeed479fc719e25ab20d0cbc5b050d78e3"
Nov 02 17:59:46.764203 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.764154    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused" start-of-body=
Nov 02 17:59:46.789975 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.764231    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": dial tcp 10.134.0.198:15021: connect: connection refused"
Nov 02 17:59:46.888514 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.888377    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:46.976871 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.965920    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:46.976871 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.966189    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 17:59:46.976871 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.966927    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:59:46.976871 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.976677    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 17:59:46.994851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.992510    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:46.994851 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:46.993688    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:59:47.349850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:47.346434    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:47.423568 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:47.423538    4456 scope.go:122] "RemoveContainer" containerID="af497034a0769c385617fd2ca325650ebb53a06c30ab8f3d7f487ba00bc37c69"
Nov 02 17:59:47.494536 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:47.493336    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:47.521427 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:47.521383    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:47.523329 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:47.523302    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:59:47.545843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:47.543452    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:47.545843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:47.544780    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:47.586629 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:47.577428    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:47.786972 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:47.625476    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 17:59:47.786972 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:47.625527    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 17:59:47.786972 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:47.728377    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 02 17:59:47.786972 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:47.728427    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 02 17:59:47.972843 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:47.966407    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:47.983956 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:47.977161    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:48.077903 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.071490    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:48.078199 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.078179    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 17:59:48.078924 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.078905    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 17:59:48.079091 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.079080    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:59:48.109116 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.109087    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:59:48.109351 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.109337    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:48.109466 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.072912    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 17:59:48.165840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.163483    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 17:59:48.187841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.178284    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:48.351550 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.349175    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 17:59:48.351550 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.350497    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 17:59:48.351550 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.350912    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 17:59:48.398550 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.398089    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 17:59:48.428324 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.428249    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:48.628898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.605354    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 17:59:48.628898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.605409    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 17:59:48.721143 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.720666    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 02 17:59:48.721143 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.720720    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 02 17:59:48.751074 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.742736    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"]
Nov 02 17:59:48.770784 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.770443    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"
Nov 02 17:59:48.838838 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.814186    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fbbbf8\""
Nov 02 17:59:48.839113 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.830685    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"]
Nov 02 17:59:48.875413 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.875386    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"]
Nov 02 17:59:48.878085 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.878047    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dpl8s\" (UniqueName: \"kubernetes.io/projected/3d398f17-735a-4d4b-b1cc-5b24fd266535-kube-api-access-dpl8s\") pod \"payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5\" (UID: \"3d398f17-735a-4d4b-b1cc-5b24fd266535\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"
Nov 02 17:59:48.878394 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.878372    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5\" (UID: \"3d398f17-735a-4d4b-b1cc-5b24fd266535\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"
Nov 02 17:59:48.942439 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.942408    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fbbbf8-dockercfg-29vsx\""
Nov 02 17:59:48.943057 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:48.943022    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"
Nov 02 17:59:49.011567 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.011528    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"]
Nov 02 17:59:49.050497 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.050451    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:49.112670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.072414    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n\" (UID: \"85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"
Nov 02 17:59:49.112670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.072617    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dpl8s\" (UniqueName: \"kubernetes.io/projected/3d398f17-735a-4d4b-b1cc-5b24fd266535-kube-api-access-dpl8s\") pod \"payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5\" (UID: \"3d398f17-735a-4d4b-b1cc-5b24fd266535\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"
Nov 02 17:59:49.112670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.072860    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5\" (UID: \"3d398f17-735a-4d4b-b1cc-5b24fd266535\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"
Nov 02 17:59:49.112670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.073011    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dv7vb\" (UniqueName: \"kubernetes.io/projected/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-kube-api-access-dv7vb\") pod \"payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n\" (UID: \"85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"
Nov 02 17:59:49.112670 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.074662    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume\") pod \"payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5\" (UID: \"3d398f17-735a-4d4b-b1cc-5b24fd266535\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"
Nov 02 17:59:49.278910 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.265717    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dv7vb\" (UniqueName: \"kubernetes.io/projected/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-kube-api-access-dv7vb\") pod \"payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n\" (UID: \"85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"
Nov 02 17:59:49.279397 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.279363    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n\" (UID: \"85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"
Nov 02 17:59:49.294889 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.293879    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume\") pod \"payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n\" (UID: \"85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"
Nov 02 17:59:49.554902 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.549634    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dv7vb\" (UniqueName: \"kubernetes.io/projected/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-kube-api-access-dv7vb\") pod \"payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n\" (UID: \"85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3\") " pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"
Nov 02 17:59:49.607226 ip-10-0-141-87 kubenswrapper[4456]: E1102 17:59:49.607188    4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/system.slice/system-systemd\\\\x2dcoredump.slice/systemd-coredump@20-33769-0.service\": RecentStats: unable to find data in memory cache]"
Nov 02 17:59:49.623515 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.623484    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"
Nov 02 17:59:49.629270 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.629244    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dpl8s\" (UniqueName: \"kubernetes.io/projected/3d398f17-735a-4d4b-b1cc-5b24fd266535-kube-api-access-dpl8s\") pod \"payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5\" (UID: \"3d398f17-735a-4d4b-b1cc-5b24fd266535\") " pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"
Nov 02 17:59:49.674488 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.674169    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 17:59:49.674488 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.674230    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 17:59:49.718850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.708082    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 02 17:59:49.718850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.708264    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 02 17:59:49.838142 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.837317    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"
Nov 02 17:59:49.965153 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.965124    4456 generic.go:356] "Generic (PLEG): container finished" podID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerID="17526f5191570a9cb7a3efb0f2691ee4078b3c90dee5e66f8d2131400d0a1ec8" exitCode=0
Nov 02 17:59:49.965330 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.965270    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" event={"ID":"67b961eb-ab19-4d8d-bb2a-db7c94a93436","Type":"ContainerDied","Data":"17526f5191570a9cb7a3efb0f2691ee4078b3c90dee5e66f8d2131400d0a1ec8"}
Nov 02 17:59:49.966161 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.966136    4456 scope.go:122] "RemoveContainer" containerID="17526f5191570a9cb7a3efb0f2691ee4078b3c90dee5e66f8d2131400d0a1ec8"
Nov 02 17:59:49.969528 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.969498    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:49.969850 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.969809    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:49.970027 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:49.970010    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 17:59:50.663847 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:50.626480    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 17:59:50.665525 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:50.665471    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 17:59:50.701051 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:50.701012    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:59:50.754970 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:50.754937    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 02 17:59:50.755205 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:50.755181    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 02 17:59:50.844577 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:50.828379    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:50.925955 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:50.925866    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"]
Nov 02 17:59:50.974293 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:50.974252    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"]
Nov 02 17:59:51.070840 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:51.068748    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" event={"ID":"67b961eb-ab19-4d8d-bb2a-db7c94a93436","Type":"ContainerStarted","Data":"f5f583c97bede29bd199be0ec2003a50248abd83bffd0425ce6e7cb495a11ca3"}
Nov 02 17:59:51.071585 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:51.071548    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" event={"ID":"3d398f17-735a-4d4b-b1cc-5b24fd266535","Type":"ContainerStarted","Data":"772c1ae8cdded18a6da829cad4beb7330175fc9c81b6e9f656c5a3b80448a335"}
Nov 02 17:59:51.074073 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:51.074047    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" event={"ID":"85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3","Type":"ContainerStarted","Data":"d719284df40bfeba040610251ea9252cec8432f5ba7b2b4f480accefeb3949d6"}
Nov 02 17:59:51.157268 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:51.157236    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:59:51.161466 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:51.160662    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 17:59:51.161466 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:51.160729    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 17:59:51.625013 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:51.624837    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused" start-of-body=
Nov 02 17:59:51.625199 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:51.625079    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": dial tcp 10.134.0.181:15021: connect: connection refused"
Nov 02 17:59:51.714114 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:51.714074    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 02 17:59:51.714725 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:51.714695    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 02 17:59:52.124842 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:52.124380    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" event={"ID":"3d398f17-735a-4d4b-b1cc-5b24fd266535","Type":"ContainerStarted","Data":"6c24194466a92255caca64b6c1b7d27bee17e48cdd5018b681fa9f4ecb1b733e"}
Nov 02 17:59:52.136837 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:52.125711    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"
Nov 02 17:59:52.150173 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:52.147380    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 17:59:52.150173 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:52.147435    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 17:59:52.150450 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:52.150194    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" event={"ID":"85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3","Type":"ContainerStarted","Data":"83a5c5da51e2a33c43fedd46148933222917fbcba3a4cacd320bc29364bbf103"}
Nov 02 17:59:52.150450 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:52.150384    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"
Nov 02 17:59:52.174509 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:52.174459    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" podStartSLOduration=4.174440705 podStartE2EDuration="4.174440705s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:52.170704293 +0000 UTC m=+1808.661767468" watchObservedRunningTime="2026-11-02 17:59:52.174440705 +0000 UTC m=+1808.665503867"
Nov 02 17:59:52.246911 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:52.243270    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" podStartSLOduration=4.243249689 podStartE2EDuration="4.243249689s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 17:59:48 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 17:59:52.231296829 +0000 UTC m=+1808.722359992" watchObservedRunningTime="2026-11-02 17:59:52.243249689 +0000 UTC m=+1808.734312851"
Nov 02 17:59:52.458844 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:52.455671    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:52.646841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:52.643854    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:59:52.646841 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:52.645585    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:59:52.669169 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:52.669142    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 17:59:52.719607 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:52.719470    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused" start-of-body=
Nov 02 17:59:52.720414 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:52.720286    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": dial tcp 10.134.0.179:15021: connect: connection refused"
Nov 02 17:59:53.147637 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:53.147552    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 17:59:53.147637 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:53.147604    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 17:59:53.723602 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:53.723565    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:59:53.724918 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:53.724894    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 17:59:54.135898 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:54.134482    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:54.151253 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:54.150307    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 17:59:54.151253 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:54.150382    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 17:59:54.279868 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:54.279798    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:59:55.147413 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:55.147370    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 17:59:55.148033 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:55.147429    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 17:59:55.726520 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:55.726482    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:56.147639 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:56.147549    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 17:59:56.147639 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:56.147606    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 17:59:57.146915 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:57.146880    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused" start-of-body=
Nov 02 17:59:57.147146 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:57.146948    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": dial tcp 10.134.0.185:15021: connect: connection refused"
Nov 02 17:59:57.245617 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:57.245584    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 17:59:58.152495 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:58.152453    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:59:58.153565 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:58.153546    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 17:59:58.902108 ip-10-0-141-87 kubenswrapper[4456]: I1102 17:59:58.902076    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:00.205951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.193120    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"]
Nov 02 18:00:00.223869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.219002    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"]
Nov 02 18:00:00.228380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.225920    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"
Nov 02 18:00:00.241452 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.241421    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"
Nov 02 18:00:00.292677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.292646    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"]
Nov 02 18:00:00.318552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.302929    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:00:00.349846 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.332939    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jd5z4\" (UniqueName: \"kubernetes.io/projected/3c1340c7-b300-40a2-a155-1f43448eccf4-kube-api-access-jd5z4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66\" (UID: \"3c1340c7-b300-40a2-a155-1f43448eccf4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"
Nov 02 18:00:00.541058 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.540970    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1340c7-b300-40a2-a155-1f43448eccf4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66\" (UID: \"3c1340c7-b300-40a2-a155-1f43448eccf4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"
Nov 02 18:00:00.587419 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.585953    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7d98v\" (UniqueName: \"kubernetes.io/projected/12b1615a-fc8e-4611-a7f7-84b25bd4295a-kube-api-access-7d98v\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b\" (UID: \"12b1615a-fc8e-4611-a7f7-84b25bd4295a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"
Nov 02 18:00:00.592225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.458362    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"
Nov 02 18:00:00.608035 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.607994    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b1615a-fc8e-4611-a7f7-84b25bd4295a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b\" (UID: \"12b1615a-fc8e-4611-a7f7-84b25bd4295a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"
Nov 02 18:00:00.642865 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.458235    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"]
Nov 02 18:00:00.873465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.869897    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:00.918663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.899695    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jd5z4\" (UniqueName: \"kubernetes.io/projected/3c1340c7-b300-40a2-a155-1f43448eccf4-kube-api-access-jd5z4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66\" (UID: \"3c1340c7-b300-40a2-a155-1f43448eccf4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"
Nov 02 18:00:00.918663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.899847    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1340c7-b300-40a2-a155-1f43448eccf4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66\" (UID: \"3c1340c7-b300-40a2-a155-1f43448eccf4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"
Nov 02 18:00:00.918663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.899886    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7d98v\" (UniqueName: \"kubernetes.io/projected/12b1615a-fc8e-4611-a7f7-84b25bd4295a-kube-api-access-7d98v\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b\" (UID: \"12b1615a-fc8e-4611-a7f7-84b25bd4295a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"
Nov 02 18:00:00.918663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.900005    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b1615a-fc8e-4611-a7f7-84b25bd4295a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b\" (UID: \"12b1615a-fc8e-4611-a7f7-84b25bd4295a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"
Nov 02 18:00:00.918663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.900203    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9c7fm\" (UniqueName: \"kubernetes.io/projected/59097f3d-9900-4383-9979-f3a60427b529-kube-api-access-9c7fm\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8\" (UID: \"59097f3d-9900-4383-9979-f3a60427b529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"
Nov 02 18:00:00.918663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.900257    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59097f3d-9900-4383-9979-f3a60427b529-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8\" (UID: \"59097f3d-9900-4383-9979-f3a60427b529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"
Nov 02 18:00:00.918663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.901395    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1340c7-b300-40a2-a155-1f43448eccf4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66\" (UID: \"3c1340c7-b300-40a2-a155-1f43448eccf4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"
Nov 02 18:00:00.918663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:00.902241    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b1615a-fc8e-4611-a7f7-84b25bd4295a-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b\" (UID: \"12b1615a-fc8e-4611-a7f7-84b25bd4295a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"
Nov 02 18:00:01.030837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.030250    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"
Nov 02 18:00:01.035500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.035471    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9c7fm\" (UniqueName: \"kubernetes.io/projected/59097f3d-9900-4383-9979-f3a60427b529-kube-api-access-9c7fm\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8\" (UID: \"59097f3d-9900-4383-9979-f3a60427b529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"
Nov 02 18:00:01.035773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.035753    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59097f3d-9900-4383-9979-f3a60427b529-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8\" (UID: \"59097f3d-9900-4383-9979-f3a60427b529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"
Nov 02 18:00:01.036875 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.036851    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59097f3d-9900-4383-9979-f3a60427b529-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8\" (UID: \"59097f3d-9900-4383-9979-f3a60427b529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"
Nov 02 18:00:01.038293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.037868    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"]
Nov 02 18:00:01.056650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.056576    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"]
Nov 02 18:00:01.090427 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.058761    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"
Nov 02 18:00:01.091153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.059575    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jd5z4\" (UniqueName: \"kubernetes.io/projected/3c1340c7-b300-40a2-a155-1f43448eccf4-kube-api-access-jd5z4\") pod \"maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66\" (UID: \"3c1340c7-b300-40a2-a155-1f43448eccf4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"
Nov 02 18:00:01.153843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.114082    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"]
Nov 02 18:00:01.153843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.127879    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7d98v\" (UniqueName: \"kubernetes.io/projected/12b1615a-fc8e-4611-a7f7-84b25bd4295a-kube-api-access-7d98v\") pod \"maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b\" (UID: \"12b1615a-fc8e-4611-a7f7-84b25bd4295a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"
Nov 02 18:00:01.177848 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.157782    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"
Nov 02 18:00:01.244849 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.237858    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"
Nov 02 18:00:01.312886 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.299500    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9c7fm\" (UniqueName: \"kubernetes.io/projected/59097f3d-9900-4383-9979-f3a60427b529-kube-api-access-9c7fm\") pod \"maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8\" (UID: \"59097f3d-9900-4383-9979-f3a60427b529\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"
Nov 02 18:00:01.312886 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.299723    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"
Nov 02 18:00:01.342922 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.342845    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"]
Nov 02 18:00:01.355652 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.354664    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536e0993-02c4-43f0-a555-3bbadd26d1ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw\" (UID: \"536e0993-02c4-43f0-a555-3bbadd26d1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"
Nov 02 18:00:01.355652 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.354879    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e4a236e-0151-4496-a90d-58cccb957e91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z\" (UID: \"7e4a236e-0151-4496-a90d-58cccb957e91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"
Nov 02 18:00:01.355652 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.354939    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5dtr\" (UniqueName: \"kubernetes.io/projected/f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab-kube-api-access-f5dtr\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd\" (UID: \"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"
Nov 02 18:00:01.355652 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.355012    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd\" (UID: \"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"
Nov 02 18:00:01.355652 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.355217    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-k69sh\" (UniqueName: \"kubernetes.io/projected/7e4a236e-0151-4496-a90d-58cccb957e91-kube-api-access-k69sh\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z\" (UID: \"7e4a236e-0151-4496-a90d-58cccb957e91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"
Nov 02 18:00:01.355652 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.355242    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ftrmx\" (UniqueName: \"kubernetes.io/projected/536e0993-02c4-43f0-a555-3bbadd26d1ab-kube-api-access-ftrmx\") pod \"maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw\" (UID: \"536e0993-02c4-43f0-a555-3bbadd26d1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"
Nov 02 18:00:01.477267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.470140    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536e0993-02c4-43f0-a555-3bbadd26d1ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw\" (UID: \"536e0993-02c4-43f0-a555-3bbadd26d1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"
Nov 02 18:00:01.477267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.470338    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e4a236e-0151-4496-a90d-58cccb957e91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z\" (UID: \"7e4a236e-0151-4496-a90d-58cccb957e91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"
Nov 02 18:00:01.477267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.470392    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5dtr\" (UniqueName: \"kubernetes.io/projected/f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab-kube-api-access-f5dtr\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd\" (UID: \"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"
Nov 02 18:00:01.477267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.470450    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd\" (UID: \"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"
Nov 02 18:00:01.477267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.470635    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-k69sh\" (UniqueName: \"kubernetes.io/projected/7e4a236e-0151-4496-a90d-58cccb957e91-kube-api-access-k69sh\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z\" (UID: \"7e4a236e-0151-4496-a90d-58cccb957e91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"
Nov 02 18:00:01.477267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.470661    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ftrmx\" (UniqueName: \"kubernetes.io/projected/536e0993-02c4-43f0-a555-3bbadd26d1ab-kube-api-access-ftrmx\") pod \"maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw\" (UID: \"536e0993-02c4-43f0-a555-3bbadd26d1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"
Nov 02 18:00:01.477267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.471896    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536e0993-02c4-43f0-a555-3bbadd26d1ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw\" (UID: \"536e0993-02c4-43f0-a555-3bbadd26d1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"
Nov 02 18:00:01.477267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.472553    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e4a236e-0151-4496-a90d-58cccb957e91-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z\" (UID: \"7e4a236e-0151-4496-a90d-58cccb957e91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"
Nov 02 18:00:01.489642 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.489573    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd\" (UID: \"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"
Nov 02 18:00:01.539311 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.532314    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"
Nov 02 18:00:01.539311 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.535528    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:01.546894 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.545971    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"]
Nov 02 18:00:01.546894 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.546018    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"]
Nov 02 18:00:01.546894 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.546049    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"]
Nov 02 18:00:01.566917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.555048    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"
Nov 02 18:00:01.895369 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.857757    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5dtr\" (UniqueName: \"kubernetes.io/projected/f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab-kube-api-access-f5dtr\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd\" (UID: \"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"
Nov 02 18:00:01.909439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:01.898192    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"
Nov 02 18:00:02.051842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:02.032215    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/661af531-d82f-434e-855d-516e8cc9caec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t\" (UID: \"661af531-d82f-434e-855d-516e8cc9caec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"
Nov 02 18:00:02.051842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:02.033105    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fvcjm\" (UniqueName: \"kubernetes.io/projected/661af531-d82f-434e-855d-516e8cc9caec-kube-api-access-fvcjm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t\" (UID: \"661af531-d82f-434e-855d-516e8cc9caec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"
Nov 02 18:00:02.153700 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:02.153615    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-k69sh\" (UniqueName: \"kubernetes.io/projected/7e4a236e-0151-4496-a90d-58cccb957e91-kube-api-access-k69sh\") pod \"maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z\" (UID: \"7e4a236e-0151-4496-a90d-58cccb957e91\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"
Nov 02 18:00:02.173007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:02.172934    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:02.289890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:02.240027    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:02.523045 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:02.507361    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:02.576117 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:02.566458    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:02.704841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:02.700305    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ftrmx\" (UniqueName: \"kubernetes.io/projected/536e0993-02c4-43f0-a555-3bbadd26d1ab-kube-api-access-ftrmx\") pod \"maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw\" (UID: \"536e0993-02c4-43f0-a555-3bbadd26d1ab\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"
Nov 02 18:00:02.704841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:02.703462    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8kgmn\" (UniqueName: \"kubernetes.io/projected/b21be16b-0021-4551-8662-a7c5fac4e7ad-kube-api-access-8kgmn\") pod \"maas-api-key-cleanup-29894040-nf8sf\" (UID: \"b21be16b-0021-4551-8662-a7c5fac4e7ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"
Nov 02 18:00:02.704841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:02.703650    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b21be16b-0021-4551-8662-a7c5fac4e7ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894040-nf8sf\" (UID: \"b21be16b-0021-4551-8662-a7c5fac4e7ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"
Nov 02 18:00:02.729322 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:02.728015    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:00:02.828894 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:02.798925    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:02.845407 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:02.845353    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:03.111917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.110192    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:03.217475 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.216657    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"
Nov 02 18:00:03.256151 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.243117    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:03.256151 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.243199    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"
Nov 02 18:00:03.256151 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.255724    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/661af531-d82f-434e-855d-516e8cc9caec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t\" (UID: \"661af531-d82f-434e-855d-516e8cc9caec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"
Nov 02 18:00:03.256151 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.255845    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fvcjm\" (UniqueName: \"kubernetes.io/projected/661af531-d82f-434e-855d-516e8cc9caec-kube-api-access-fvcjm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t\" (UID: \"661af531-d82f-434e-855d-516e8cc9caec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"
Nov 02 18:00:03.256151 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.255948    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8kgmn\" (UniqueName: \"kubernetes.io/projected/b21be16b-0021-4551-8662-a7c5fac4e7ad-kube-api-access-8kgmn\") pod \"maas-api-key-cleanup-29894040-nf8sf\" (UID: \"b21be16b-0021-4551-8662-a7c5fac4e7ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"
Nov 02 18:00:03.256878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.256709    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b21be16b-0021-4551-8662-a7c5fac4e7ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894040-nf8sf\" (UID: \"b21be16b-0021-4551-8662-a7c5fac4e7ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"
Nov 02 18:00:03.259540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.258116    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b21be16b-0021-4551-8662-a7c5fac4e7ad-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894040-nf8sf\" (UID: \"b21be16b-0021-4551-8662-a7c5fac4e7ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"
Nov 02 18:00:03.259540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.258949    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/661af531-d82f-434e-855d-516e8cc9caec-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t\" (UID: \"661af531-d82f-434e-855d-516e8cc9caec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"
Nov 02 18:00:03.354181 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:03.301586    4456 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.551s"
Nov 02 18:00:03.354181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.326468    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"
Nov 02 18:00:03.354181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.326896    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 18:00:03.354181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.327170    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 18:00:03.354181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.327200    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"]
Nov 02 18:00:03.463732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.457532    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8kgmn\" (UniqueName: \"kubernetes.io/projected/b21be16b-0021-4551-8662-a7c5fac4e7ad-kube-api-access-8kgmn\") pod \"maas-api-key-cleanup-29894040-nf8sf\" (UID: \"b21be16b-0021-4551-8662-a7c5fac4e7ad\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"
Nov 02 18:00:03.525227 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.498206    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:03.566885 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.566619    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:03.636621 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.578553    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"
Nov 02 18:00:03.636621 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.578616    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"]
Nov 02 18:00:03.636621 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.606646    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"
Nov 02 18:00:03.636621 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.634950    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9\" (UID: \"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"
Nov 02 18:00:03.636621 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.635187    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-znnn9\" (UniqueName: \"kubernetes.io/projected/2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211-kube-api-access-znnn9\") pod \"maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9\" (UID: \"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"
Nov 02 18:00:03.643713 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.643521    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"
Nov 02 18:00:03.659639 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.653641    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fvcjm\" (UniqueName: \"kubernetes.io/projected/661af531-d82f-434e-855d-516e8cc9caec-kube-api-access-fvcjm\") pod \"maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t\" (UID: \"661af531-d82f-434e-855d-516e8cc9caec\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"
Nov 02 18:00:03.659639 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.657554    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"
Nov 02 18:00:03.768886 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.755363    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9\" (UID: \"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"
Nov 02 18:00:03.768886 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.755594    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-znnn9\" (UniqueName: \"kubernetes.io/projected/2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211-kube-api-access-znnn9\") pod \"maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9\" (UID: \"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"
Nov 02 18:00:03.768886 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.757059    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9\" (UID: \"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"
Nov 02 18:00:03.768886 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.761839    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"
Nov 02 18:00:03.853455 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.853386    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-znnn9\" (UniqueName: \"kubernetes.io/projected/2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211-kube-api-access-znnn9\") pod \"maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9\" (UID: \"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"
Nov 02 18:00:03.856853 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.856474    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"]
Nov 02 18:00:03.858094 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.857413    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bm55m\" (UniqueName: \"kubernetes.io/projected/30b7824a-62b5-4395-8980-2d38452785a7-kube-api-access-bm55m\") pod \"maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc\" (UID: \"30b7824a-62b5-4395-8980-2d38452785a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"
Nov 02 18:00:03.858094 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.857901    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b7824a-62b5-4395-8980-2d38452785a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc\" (UID: \"30b7824a-62b5-4395-8980-2d38452785a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"
Nov 02 18:00:03.867363 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.867339    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"]
Nov 02 18:00:03.867600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.867582    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"
Nov 02 18:00:03.873809 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.873785    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:00:03.873936 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.873843    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"]
Nov 02 18:00:03.873936 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.873874    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"]
Nov 02 18:00:03.873936 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.873902    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 18:00:03.873936 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.873935    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 18:00:03.874953 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.873963    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:00:03.874953 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.873976    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"]
Nov 02 18:00:03.897410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.878198    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"
Nov 02 18:00:03.923419 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.921950    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"]
Nov 02 18:00:03.931196 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.927892    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.961418    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b7824a-62b5-4395-8980-2d38452785a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc\" (UID: \"30b7824a-62b5-4395-8980-2d38452785a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.961448    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"]
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.961475    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9dv4n\" (UniqueName: \"kubernetes.io/projected/8765d0ad-655c-4f40-8825-5e0092d2c0aa-kube-api-access-9dv4n\") pod \"maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc\" (UID: \"8765d0ad-655c-4f40-8825-5e0092d2c0aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.961493    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66" event={"ID":"3c1340c7-b300-40a2-a155-1f43448eccf4","Type":"ContainerStarted","Data":"4b63477e8e5de8652303f9371bb54b74a35b29b5cef59ee69802402b8c145acb"}
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.961532    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bm55m\" (UniqueName: \"kubernetes.io/projected/30b7824a-62b5-4395-8980-2d38452785a7-kube-api-access-bm55m\") pod \"maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc\" (UID: \"30b7824a-62b5-4395-8980-2d38452785a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.961538    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"]
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.961596    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"]
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.961625    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"]
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.961652    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"]
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.961687    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"]
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.961713    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"]
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.961713    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8765d0ad-655c-4f40-8825-5e0092d2c0aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc\" (UID: \"8765d0ad-655c-4f40-8825-5e0092d2c0aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.961760    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"]
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.961785    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"]
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.961999    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"]
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.962760    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b7824a-62b5-4395-8980-2d38452785a7-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc\" (UID: \"30b7824a-62b5-4395-8980-2d38452785a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"
Nov 02 18:00:03.966459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:03.963179    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"
Nov 02 18:00:04.036525 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.036496    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"
Nov 02 18:00:04.066284 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.065580    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9dv4n\" (UniqueName: \"kubernetes.io/projected/8765d0ad-655c-4f40-8825-5e0092d2c0aa-kube-api-access-9dv4n\") pod \"maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc\" (UID: \"8765d0ad-655c-4f40-8825-5e0092d2c0aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"
Nov 02 18:00:04.066634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.066609    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6kxls\" (UniqueName: \"kubernetes.io/projected/aba740c9-0c49-4641-83db-41bdbf8b9794-kube-api-access-6kxls\") pod \"maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h\" (UID: \"aba740c9-0c49-4641-83db-41bdbf8b9794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"
Nov 02 18:00:04.066878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.066859    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h\" (UID: \"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"
Nov 02 18:00:04.067075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.067056    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8765d0ad-655c-4f40-8825-5e0092d2c0aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc\" (UID: \"8765d0ad-655c-4f40-8825-5e0092d2c0aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"
Nov 02 18:00:04.067223 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.067208    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-825ks\" (UniqueName: \"kubernetes.io/projected/a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c-kube-api-access-825ks\") pod \"maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h\" (UID: \"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"
Nov 02 18:00:04.067353 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.067339    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-m7qxx\" (UniqueName: \"kubernetes.io/projected/40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4-kube-api-access-m7qxx\") pod \"maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5\" (UID: \"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"
Nov 02 18:00:04.067599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.067521    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5\" (UID: \"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"
Nov 02 18:00:04.067732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.067717    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba740c9-0c49-4641-83db-41bdbf8b9794-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h\" (UID: \"aba740c9-0c49-4641-83db-41bdbf8b9794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"
Nov 02 18:00:04.069100 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.069080    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8765d0ad-655c-4f40-8825-5e0092d2c0aa-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc\" (UID: \"8765d0ad-655c-4f40-8825-5e0092d2c0aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"
Nov 02 18:00:04.148510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.136383    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"]
Nov 02 18:00:04.149620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.149439    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bm55m\" (UniqueName: \"kubernetes.io/projected/30b7824a-62b5-4395-8980-2d38452785a7-kube-api-access-bm55m\") pod \"maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc\" (UID: \"30b7824a-62b5-4395-8980-2d38452785a7\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"
Nov 02 18:00:04.179183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.176693    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6kxls\" (UniqueName: \"kubernetes.io/projected/aba740c9-0c49-4641-83db-41bdbf8b9794-kube-api-access-6kxls\") pod \"maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h\" (UID: \"aba740c9-0c49-4641-83db-41bdbf8b9794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"
Nov 02 18:00:04.179183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.176776    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h\" (UID: \"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"
Nov 02 18:00:04.179183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.176875    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-825ks\" (UniqueName: \"kubernetes.io/projected/a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c-kube-api-access-825ks\") pod \"maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h\" (UID: \"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"
Nov 02 18:00:04.190473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.176923    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-m7qxx\" (UniqueName: \"kubernetes.io/projected/40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4-kube-api-access-m7qxx\") pod \"maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5\" (UID: \"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"
Nov 02 18:00:04.190473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.189590    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5\" (UID: \"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"
Nov 02 18:00:04.190473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.189648    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba740c9-0c49-4641-83db-41bdbf8b9794-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h\" (UID: \"aba740c9-0c49-4641-83db-41bdbf8b9794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"
Nov 02 18:00:04.196850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.196664    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba740c9-0c49-4641-83db-41bdbf8b9794-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h\" (UID: \"aba740c9-0c49-4641-83db-41bdbf8b9794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"
Nov 02 18:00:04.196850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.177888    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h\" (UID: \"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"
Nov 02 18:00:04.212499 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.210251    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5\" (UID: \"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"
Nov 02 18:00:04.212499 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.210972    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9dv4n\" (UniqueName: \"kubernetes.io/projected/8765d0ad-655c-4f40-8825-5e0092d2c0aa-kube-api-access-9dv4n\") pod \"maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc\" (UID: \"8765d0ad-655c-4f40-8825-5e0092d2c0aa\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"
Nov 02 18:00:04.237611 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.232506    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6kxls\" (UniqueName: \"kubernetes.io/projected/aba740c9-0c49-4641-83db-41bdbf8b9794-kube-api-access-6kxls\") pod \"maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h\" (UID: \"aba740c9-0c49-4641-83db-41bdbf8b9794\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"
Nov 02 18:00:04.258643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.258613    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"
Nov 02 18:00:04.266036 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.265523    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-825ks\" (UniqueName: \"kubernetes.io/projected/a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c-kube-api-access-825ks\") pod \"maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h\" (UID: \"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"
Nov 02 18:00:04.267276 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.267165    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-m7qxx\" (UniqueName: \"kubernetes.io/projected/40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4-kube-api-access-m7qxx\") pod \"maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5\" (UID: \"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"
Nov 02 18:00:04.270776 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.270751    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"
Nov 02 18:00:04.317153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.317090    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"
Nov 02 18:00:04.331195 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.331168    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"
Nov 02 18:00:04.378838 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.368184    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"
Nov 02 18:00:04.555343 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.552432    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"
Nov 02 18:00:04.556399 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.479516    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"]
Nov 02 18:00:04.844014 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.843924    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"]
Nov 02 18:00:04.987272 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:04.987057    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:05.074618 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:05.073469    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8" event={"ID":"59097f3d-9900-4383-9979-f3a60427b529","Type":"ContainerStarted","Data":"1819ed936b96c80e788150ad0269c21e8586afae6451f67721bba2ff8f842cdd"}
Nov 02 18:00:05.099180 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:05.099099    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd" event={"ID":"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab","Type":"ContainerStarted","Data":"ce601640105cce7c7cf188e745d7e0cdd0b89d7bbb3e15e6fa3f0fbb203f7161"}
Nov 02 18:00:05.121059 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:05.120984    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b" event={"ID":"12b1615a-fc8e-4611-a7f7-84b25bd4295a","Type":"ContainerStarted","Data":"d5750c3d3236c76e8788ab707585fcf1e15251331e22549c9c7862f049daf334"}
Nov 02 18:00:05.178727 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:05.166978    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"]
Nov 02 18:00:05.254845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:05.213148    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66" podStartSLOduration=5.213120606 podStartE2EDuration="5.213120606s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:05.151153895 +0000 UTC m=+1821.642217058" watchObservedRunningTime="2026-11-02 18:00:05.213120606 +0000 UTC m=+1821.704183768"
Nov 02 18:00:05.669836 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:05.669572    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"]
Nov 02 18:00:05.669836 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:05.669637    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"]
Nov 02 18:00:05.951411 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:05.951383    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"]
Nov 02 18:00:05.955635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:05.955490    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"]
Nov 02 18:00:05.961101 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:00:05.961073    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8765d0ad_655c_4f40_8825_5e0092d2c0aa.slice/crio-9f92b2913357352293c1e1a9e092c50277168f8b6a38adb51fa69ea492f6a8d4 WatchSource:0}: Error finding container 9f92b2913357352293c1e1a9e092c50277168f8b6a38adb51fa69ea492f6a8d4: Status 404 returned error can't find the container with id 9f92b2913357352293c1e1a9e092c50277168f8b6a38adb51fa69ea492f6a8d4
Nov 02 18:00:05.963951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:05.963884    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"]
Nov 02 18:00:06.042663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.042398    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"]
Nov 02 18:00:06.042663 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:00:06.042638    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod661af531_d82f_434e_855d_516e8cc9caec.slice/crio-185b56cb7b78f3b3062351903a83fe8d409c4b3c943dbfddc03f90b1b5c49136 WatchSource:0}: Error finding container 185b56cb7b78f3b3062351903a83fe8d409c4b3c943dbfddc03f90b1b5c49136: Status 404 returned error can't find the container with id 185b56cb7b78f3b3062351903a83fe8d409c4b3c943dbfddc03f90b1b5c49136
Nov 02 18:00:06.045739 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:00:06.045377    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod30b7824a_62b5_4395_8980_2d38452785a7.slice/crio-f7d2b4f4666e90121df372558ff656b64536a5cd2acb020d7877969a2df95cc6 WatchSource:0}: Error finding container f7d2b4f4666e90121df372558ff656b64536a5cd2acb020d7877969a2df95cc6: Status 404 returned error can't find the container with id f7d2b4f4666e90121df372558ff656b64536a5cd2acb020d7877969a2df95cc6
Nov 02 18:00:06.051230 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:00:06.051203    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podaba740c9_0c49_4641_83db_41bdbf8b9794.slice/crio-230ceb62f1f8502bb24ee3a71890b4392571f9efdfee91fb07b88e7f7d940359 WatchSource:0}: Error finding container 230ceb62f1f8502bb24ee3a71890b4392571f9efdfee91fb07b88e7f7d940359: Status 404 returned error can't find the container with id 230ceb62f1f8502bb24ee3a71890b4392571f9efdfee91fb07b88e7f7d940359
Nov 02 18:00:06.051388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.051367    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"]
Nov 02 18:00:06.054429 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:00:06.054406    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod40181b2b_2dd4_4f0b_bbcd_eccdc2a4c9b4.slice/crio-343b880cd19d87c1e057e7cf12151a7e9ba78b045e26d59e96f1c6e000e40a23 WatchSource:0}: Error finding container 343b880cd19d87c1e057e7cf12151a7e9ba78b045e26d59e96f1c6e000e40a23: Status 404 returned error can't find the container with id 343b880cd19d87c1e057e7cf12151a7e9ba78b045e26d59e96f1c6e000e40a23
Nov 02 18:00:06.063363 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.063262    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"]
Nov 02 18:00:06.068443 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.068412    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"]
Nov 02 18:00:06.143557 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.143458    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc" event={"ID":"30b7824a-62b5-4395-8980-2d38452785a7","Type":"ContainerStarted","Data":"f7d2b4f4666e90121df372558ff656b64536a5cd2acb020d7877969a2df95cc6"}
Nov 02 18:00:06.145660 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.145626    4456 generic.go:356] "Generic (PLEG): container finished" podID="59097f3d-9900-4383-9979-f3a60427b529" containerID="a92b159d78b586d3550d900d76987b180efa0f579f932dfe356104fdf66861c8" exitCode=60
Nov 02 18:00:06.145884 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.145717    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8" event={"ID":"59097f3d-9900-4383-9979-f3a60427b529","Type":"ContainerDied","Data":"a92b159d78b586d3550d900d76987b180efa0f579f932dfe356104fdf66861c8"}
Nov 02 18:00:06.146102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.146082    4456 scope.go:122] "RemoveContainer" containerID="a92b159d78b586d3550d900d76987b180efa0f579f932dfe356104fdf66861c8"
Nov 02 18:00:06.149141 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.149061    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t" event={"ID":"661af531-d82f-434e-855d-516e8cc9caec","Type":"ContainerStarted","Data":"185b56cb7b78f3b3062351903a83fe8d409c4b3c943dbfddc03f90b1b5c49136"}
Nov 02 18:00:06.151359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.151331    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5" event={"ID":"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4","Type":"ContainerStarted","Data":"343b880cd19d87c1e057e7cf12151a7e9ba78b045e26d59e96f1c6e000e40a23"}
Nov 02 18:00:06.152967 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.152942    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf" event={"ID":"b21be16b-0021-4551-8662-a7c5fac4e7ad","Type":"ContainerStarted","Data":"b4c31a23f43cd47f4e4693ac8bbb9200dc93c44cb5f779d3eff934f36b5031db"}
Nov 02 18:00:06.154504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.154484    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc" event={"ID":"8765d0ad-655c-4f40-8825-5e0092d2c0aa","Type":"ContainerStarted","Data":"9f92b2913357352293c1e1a9e092c50277168f8b6a38adb51fa69ea492f6a8d4"}
Nov 02 18:00:06.156392 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.156372    4456 generic.go:356] "Generic (PLEG): container finished" podID="3c1340c7-b300-40a2-a155-1f43448eccf4" containerID="bfdcc48b236f604ccfa345139725c90caec45907a37cc8499720ea67c4e0e2ae" exitCode=60
Nov 02 18:00:06.156476 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.156449    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66" event={"ID":"3c1340c7-b300-40a2-a155-1f43448eccf4","Type":"ContainerDied","Data":"bfdcc48b236f604ccfa345139725c90caec45907a37cc8499720ea67c4e0e2ae"}
Nov 02 18:00:06.156938 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.156919    4456 scope.go:122] "RemoveContainer" containerID="bfdcc48b236f604ccfa345139725c90caec45907a37cc8499720ea67c4e0e2ae"
Nov 02 18:00:06.157934 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.157907    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9" event={"ID":"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211","Type":"ContainerStarted","Data":"cbf78392c72a42f1dd81e3ae5e1529bdeeff125fabcc90ae82dfaf6891cdca30"}
Nov 02 18:00:06.159209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.159193    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw" event={"ID":"536e0993-02c4-43f0-a555-3bbadd26d1ab","Type":"ContainerStarted","Data":"5175af497052a3a34468991569a3c6ea24bc3a0b51ccdedddb9c1b5c8a2c159e"}
Nov 02 18:00:06.162125 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.162102    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd" event={"ID":"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab","Type":"ContainerStarted","Data":"00af0641b894f15ac2392bec7831e76c32daa4650b2e69d0afd1174f969c8554"}
Nov 02 18:00:06.164873 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.164853    4456 generic.go:356] "Generic (PLEG): container finished" podID="12b1615a-fc8e-4611-a7f7-84b25bd4295a" containerID="628ca7a2e8980aa24d183b83612ae9d178b924cee1f0f6f9a1d8c0d2c07d41a4" exitCode=60
Nov 02 18:00:06.165125 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.165065    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b" event={"ID":"12b1615a-fc8e-4611-a7f7-84b25bd4295a","Type":"ContainerDied","Data":"628ca7a2e8980aa24d183b83612ae9d178b924cee1f0f6f9a1d8c0d2c07d41a4"}
Nov 02 18:00:06.165273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.165249    4456 scope.go:122] "RemoveContainer" containerID="628ca7a2e8980aa24d183b83612ae9d178b924cee1f0f6f9a1d8c0d2c07d41a4"
Nov 02 18:00:06.166992 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.166969    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z" event={"ID":"7e4a236e-0151-4496-a90d-58cccb957e91","Type":"ContainerStarted","Data":"4c3e7fe0716b1a0089dbad13de3e7165eac00a47f7fa46f97510d02003fdf244"}
Nov 02 18:00:06.167052 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.167013    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z" event={"ID":"7e4a236e-0151-4496-a90d-58cccb957e91","Type":"ContainerStarted","Data":"69a10e840ec0ca789f01232cf891d5851370f482a3b5c78461158244fb89a74f"}
Nov 02 18:00:06.168826 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.168791    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h" event={"ID":"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c","Type":"ContainerStarted","Data":"3be08b1c5df0cbd717c1d7a070a20235692993256920ea45b832148a5c4e1b2a"}
Nov 02 18:00:06.170454 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.170417    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h" event={"ID":"aba740c9-0c49-4641-83db-41bdbf8b9794","Type":"ContainerStarted","Data":"230ceb62f1f8502bb24ee3a71890b4392571f9efdfee91fb07b88e7f7d940359"}
Nov 02 18:00:06.406393 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.406329    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd" podStartSLOduration=6.406299144 podStartE2EDuration="6.406299144s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:06.360107368 +0000 UTC m=+1822.851170529" watchObservedRunningTime="2026-11-02 18:00:06.406299144 +0000 UTC m=+1822.897362306"
Nov 02 18:00:06.461921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.461848    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z" podStartSLOduration=6.461780434 podStartE2EDuration="6.461780434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:06.45212687 +0000 UTC m=+1822.943190044" watchObservedRunningTime="2026-11-02 18:00:06.461780434 +0000 UTC m=+1822.952843595"
Nov 02 18:00:06.695132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:06.669684    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:07.228884 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.226073    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9" event={"ID":"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211","Type":"ContainerStarted","Data":"ffc33a8c2309405b44408086858e196af859d6ef8b95e2bdb1ccdd2bd32ec1d2"}
Nov 02 18:00:07.228884 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.228546    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw" event={"ID":"536e0993-02c4-43f0-a555-3bbadd26d1ab","Type":"ContainerStarted","Data":"52d18f2b9a2609d93fd4f7c5f0accfb026962f48c3f39d7464b1cf9cdc97d6d9"}
Nov 02 18:00:07.231485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.231418    4456 generic.go:356] "Generic (PLEG): container finished" podID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab" containerID="00af0641b894f15ac2392bec7831e76c32daa4650b2e69d0afd1174f969c8554" exitCode=60
Nov 02 18:00:07.231648 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.231548    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd" event={"ID":"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab","Type":"ContainerDied","Data":"00af0641b894f15ac2392bec7831e76c32daa4650b2e69d0afd1174f969c8554"}
Nov 02 18:00:07.233460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.233442    4456 scope.go:122] "RemoveContainer" containerID="00af0641b894f15ac2392bec7831e76c32daa4650b2e69d0afd1174f969c8554"
Nov 02 18:00:07.233938 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.233914    4456 generic.go:356] "Generic (PLEG): container finished" podID="7e4a236e-0151-4496-a90d-58cccb957e91" containerID="4c3e7fe0716b1a0089dbad13de3e7165eac00a47f7fa46f97510d02003fdf244" exitCode=60
Nov 02 18:00:07.234064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.233993    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z" event={"ID":"7e4a236e-0151-4496-a90d-58cccb957e91","Type":"ContainerDied","Data":"4c3e7fe0716b1a0089dbad13de3e7165eac00a47f7fa46f97510d02003fdf244"}
Nov 02 18:00:07.234261 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.234242    4456 scope.go:122] "RemoveContainer" containerID="4c3e7fe0716b1a0089dbad13de3e7165eac00a47f7fa46f97510d02003fdf244"
Nov 02 18:00:07.236761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.236740    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h" event={"ID":"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c","Type":"ContainerStarted","Data":"e5ceee86e2902811a1c2c933221cdbedfa5d44b83339c57136aacd58b08fb2b9"}
Nov 02 18:00:07.240537 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.239924    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc" event={"ID":"30b7824a-62b5-4395-8980-2d38452785a7","Type":"ContainerStarted","Data":"54683bcc0e9c4b74a85a536b2d2a7355cc9decd07ff18ee37f8253e77f379c52"}
Nov 02 18:00:07.241760 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.241729    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t" event={"ID":"661af531-d82f-434e-855d-516e8cc9caec","Type":"ContainerStarted","Data":"57846e6a62119a75db87ad23bc373079c09b2e8750f34250c2b93af806a6f358"}
Nov 02 18:00:07.244361 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.244334    4456 generic.go:356] "Generic (PLEG): container finished" podID="b21be16b-0021-4551-8662-a7c5fac4e7ad" containerID="e87e641d6e66ef09dde19101a44ef895c8587bd1e77ee2dd90cc0745f1dfdaeb" exitCode=60
Nov 02 18:00:07.244457 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.244418    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf" event={"ID":"b21be16b-0021-4551-8662-a7c5fac4e7ad","Type":"ContainerDied","Data":"e87e641d6e66ef09dde19101a44ef895c8587bd1e77ee2dd90cc0745f1dfdaeb"}
Nov 02 18:00:07.244661 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.244639    4456 scope.go:122] "RemoveContainer" containerID="e87e641d6e66ef09dde19101a44ef895c8587bd1e77ee2dd90cc0745f1dfdaeb"
Nov 02 18:00:07.246628 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.246604    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc" event={"ID":"8765d0ad-655c-4f40-8825-5e0092d2c0aa","Type":"ContainerStarted","Data":"46204cce816e9c8693ef3cff1a6e3f2a4a7281fa4db96873dff559d0333fb6f8"}
Nov 02 18:00:07.274106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.274063    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9" podStartSLOduration=7.274044524 podStartE2EDuration="7.274044524s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:07.26889151 +0000 UTC m=+1823.759954672" watchObservedRunningTime="2026-11-02 18:00:07.274044524 +0000 UTC m=+1823.765107687"
Nov 02 18:00:07.463049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.462095    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h" podStartSLOduration=7.46207818 podStartE2EDuration="7.46207818s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:07.461079731 +0000 UTC m=+1823.952142889" watchObservedRunningTime="2026-11-02 18:00:07.46207818 +0000 UTC m=+1823.953141344"
Nov 02 18:00:07.531280 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.531176    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc" podStartSLOduration=7.531159782 podStartE2EDuration="7.531159782s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:07.524416443 +0000 UTC m=+1824.015479605" watchObservedRunningTime="2026-11-02 18:00:07.531159782 +0000 UTC m=+1824.022222946"
Nov 02 18:00:07.578249 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.577962    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc" podStartSLOduration=7.577946508 podStartE2EDuration="7.577946508s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:07.572118275 +0000 UTC m=+1824.063181471" watchObservedRunningTime="2026-11-02 18:00:07.577946508 +0000 UTC m=+1824.069009710"
Nov 02 18:00:07.674204 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:07.674152    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t" podStartSLOduration=7.674138356 podStartE2EDuration="7.674138356s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:07.671986302 +0000 UTC m=+1824.163049476" watchObservedRunningTime="2026-11-02 18:00:07.674138356 +0000 UTC m=+1824.165201516"
Nov 02 18:00:08.259485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.259445    4456 generic.go:356] "Generic (PLEG): container finished" podID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211" containerID="ffc33a8c2309405b44408086858e196af859d6ef8b95e2bdb1ccdd2bd32ec1d2" exitCode=60
Nov 02 18:00:08.259988 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.259547    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9" event={"ID":"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211","Type":"ContainerDied","Data":"ffc33a8c2309405b44408086858e196af859d6ef8b95e2bdb1ccdd2bd32ec1d2"}
Nov 02 18:00:08.259988 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.259968    4456 scope.go:122] "RemoveContainer" containerID="ffc33a8c2309405b44408086858e196af859d6ef8b95e2bdb1ccdd2bd32ec1d2"
Nov 02 18:00:08.261901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.261860    4456 generic.go:356] "Generic (PLEG): container finished" podID="536e0993-02c4-43f0-a555-3bbadd26d1ab" containerID="52d18f2b9a2609d93fd4f7c5f0accfb026962f48c3f39d7464b1cf9cdc97d6d9" exitCode=60
Nov 02 18:00:08.262874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.261955    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw" event={"ID":"536e0993-02c4-43f0-a555-3bbadd26d1ab","Type":"ContainerDied","Data":"52d18f2b9a2609d93fd4f7c5f0accfb026962f48c3f39d7464b1cf9cdc97d6d9"}
Nov 02 18:00:08.262874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.262267    4456 scope.go:122] "RemoveContainer" containerID="52d18f2b9a2609d93fd4f7c5f0accfb026962f48c3f39d7464b1cf9cdc97d6d9"
Nov 02 18:00:08.318802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.318758    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd" event={"ID":"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab","Type":"ContainerStarted","Data":"d2b1d035332e9a95929d4ae6d5b0f6a6b0c668b0b33817b5fad7870e77591d23"}
Nov 02 18:00:08.327634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.327571    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw" podStartSLOduration=8.327550346 podStartE2EDuration="8.327550346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:07.722959409 +0000 UTC m=+1824.214022580" watchObservedRunningTime="2026-11-02 18:00:08.327550346 +0000 UTC m=+1824.818613514"
Nov 02 18:00:08.329593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.329558    4456 generic.go:356] "Generic (PLEG): container finished" podID="12b1615a-fc8e-4611-a7f7-84b25bd4295a" containerID="af88634edc19d3340f60eeabe5e9dc057f96b50fa23d222223800e3941efd065" exitCode=60
Nov 02 18:00:08.329946 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.329918    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b" event={"ID":"12b1615a-fc8e-4611-a7f7-84b25bd4295a","Type":"ContainerDied","Data":"af88634edc19d3340f60eeabe5e9dc057f96b50fa23d222223800e3941efd065"}
Nov 02 18:00:08.330069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.329987    4456 scope.go:122] "RemoveContainer" containerID="628ca7a2e8980aa24d183b83612ae9d178b924cee1f0f6f9a1d8c0d2c07d41a4"
Nov 02 18:00:08.334090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.334064    4456 scope.go:122] "RemoveContainer" containerID="af88634edc19d3340f60eeabe5e9dc057f96b50fa23d222223800e3941efd065"
Nov 02 18:00:08.334682 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:08.334652    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b_odh-ai-gateway-infra(12b1615a-fc8e-4611-a7f7-84b25bd4295a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b" podUID="12b1615a-fc8e-4611-a7f7-84b25bd4295a"
Nov 02 18:00:08.383326 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.383198    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:08.420383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.420329    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z" event={"ID":"7e4a236e-0151-4496-a90d-58cccb957e91","Type":"ContainerStarted","Data":"f3a857086d513d872f21cdf2f22f5ca35db18f8fd0324054f7e804ca700576b6"}
Nov 02 18:00:08.435422 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.435391    4456 generic.go:356] "Generic (PLEG): container finished" podID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" containerID="e5ceee86e2902811a1c2c933221cdbedfa5d44b83339c57136aacd58b08fb2b9" exitCode=60
Nov 02 18:00:08.435790 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.435737    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h" event={"ID":"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c","Type":"ContainerDied","Data":"e5ceee86e2902811a1c2c933221cdbedfa5d44b83339c57136aacd58b08fb2b9"}
Nov 02 18:00:08.436309 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.436289    4456 scope.go:122] "RemoveContainer" containerID="e5ceee86e2902811a1c2c933221cdbedfa5d44b83339c57136aacd58b08fb2b9"
Nov 02 18:00:08.442628 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.442604    4456 generic.go:356] "Generic (PLEG): container finished" podID="aba740c9-0c49-4641-83db-41bdbf8b9794" containerID="1557d8867d2ea3eeca999aed1b45377cf6be9b579cf51e8d96f47f7c485b90f8" exitCode=60
Nov 02 18:00:08.445376 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.442781    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h" event={"ID":"aba740c9-0c49-4641-83db-41bdbf8b9794","Type":"ContainerDied","Data":"1557d8867d2ea3eeca999aed1b45377cf6be9b579cf51e8d96f47f7c485b90f8"}
Nov 02 18:00:08.445376 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.443174    4456 scope.go:122] "RemoveContainer" containerID="1557d8867d2ea3eeca999aed1b45377cf6be9b579cf51e8d96f47f7c485b90f8"
Nov 02 18:00:08.457658 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.457627    4456 generic.go:356] "Generic (PLEG): container finished" podID="30b7824a-62b5-4395-8980-2d38452785a7" containerID="54683bcc0e9c4b74a85a536b2d2a7355cc9decd07ff18ee37f8253e77f379c52" exitCode=60
Nov 02 18:00:08.457951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.457925    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc" event={"ID":"30b7824a-62b5-4395-8980-2d38452785a7","Type":"ContainerDied","Data":"54683bcc0e9c4b74a85a536b2d2a7355cc9decd07ff18ee37f8253e77f379c52"}
Nov 02 18:00:08.458629 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.458607    4456 scope.go:122] "RemoveContainer" containerID="54683bcc0e9c4b74a85a536b2d2a7355cc9decd07ff18ee37f8253e77f379c52"
Nov 02 18:00:08.489654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.477507    4456 generic.go:356] "Generic (PLEG): container finished" podID="59097f3d-9900-4383-9979-f3a60427b529" containerID="5f9a879d39b89e4236b9eb6225ce16584c956f7cd103cbc02ec62a4803cc1193" exitCode=60
Nov 02 18:00:08.489654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.477601    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8" event={"ID":"59097f3d-9900-4383-9979-f3a60427b529","Type":"ContainerDied","Data":"5f9a879d39b89e4236b9eb6225ce16584c956f7cd103cbc02ec62a4803cc1193"}
Nov 02 18:00:08.489654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.481852    4456 generic.go:356] "Generic (PLEG): container finished" podID="661af531-d82f-434e-855d-516e8cc9caec" containerID="57846e6a62119a75db87ad23bc373079c09b2e8750f34250c2b93af806a6f358" exitCode=60
Nov 02 18:00:08.489654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.486387    4456 generic.go:356] "Generic (PLEG): container finished" podID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4" containerID="10c7420291b19f55707bbc27977302ab54ccad3f97bdc7f028ada7e8737c5db0" exitCode=60
Nov 02 18:00:08.490433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.490401    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t" event={"ID":"661af531-d82f-434e-855d-516e8cc9caec","Type":"ContainerDied","Data":"57846e6a62119a75db87ad23bc373079c09b2e8750f34250c2b93af806a6f358"}
Nov 02 18:00:08.490662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.490628    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5" event={"ID":"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4","Type":"ContainerDied","Data":"10c7420291b19f55707bbc27977302ab54ccad3f97bdc7f028ada7e8737c5db0"}
Nov 02 18:00:08.491678 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.490547    4456 scope.go:122] "RemoveContainer" containerID="a92b159d78b586d3550d900d76987b180efa0f579f932dfe356104fdf66861c8"
Nov 02 18:00:08.499781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.492013    4456 scope.go:122] "RemoveContainer" containerID="5f9a879d39b89e4236b9eb6225ce16584c956f7cd103cbc02ec62a4803cc1193"
Nov 02 18:00:08.499781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.492286    4456 scope.go:122] "RemoveContainer" containerID="57846e6a62119a75db87ad23bc373079c09b2e8750f34250c2b93af806a6f358"
Nov 02 18:00:08.499781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.493425    4456 scope.go:122] "RemoveContainer" containerID="10c7420291b19f55707bbc27977302ab54ccad3f97bdc7f028ada7e8737c5db0"
Nov 02 18:00:08.499781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.497344    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf" event={"ID":"b21be16b-0021-4551-8662-a7c5fac4e7ad","Type":"ContainerStarted","Data":"75c8a99bc20c57716a347ea6decc0f470c63157109103a03bfd40d533447211c"}
Nov 02 18:00:08.499781 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:08.498348    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8_odh-ai-gateway-infra(59097f3d-9900-4383-9979-f3a60427b529)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8" podUID="59097f3d-9900-4383-9979-f3a60427b529"
Nov 02 18:00:08.508600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.502571    4456 generic.go:356] "Generic (PLEG): container finished" podID="8765d0ad-655c-4f40-8825-5e0092d2c0aa" containerID="46204cce816e9c8693ef3cff1a6e3f2a4a7281fa4db96873dff559d0333fb6f8" exitCode=60
Nov 02 18:00:08.508600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.502738    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc" event={"ID":"8765d0ad-655c-4f40-8825-5e0092d2c0aa","Type":"ContainerDied","Data":"46204cce816e9c8693ef3cff1a6e3f2a4a7281fa4db96873dff559d0333fb6f8"}
Nov 02 18:00:08.508600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.503242    4456 scope.go:122] "RemoveContainer" containerID="46204cce816e9c8693ef3cff1a6e3f2a4a7281fa4db96873dff559d0333fb6f8"
Nov 02 18:00:08.511461 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.511278    4456 generic.go:356] "Generic (PLEG): container finished" podID="3c1340c7-b300-40a2-a155-1f43448eccf4" containerID="f92b5f0b99c3730fcaff1639751164dd2a42d2277b8537a3af06737225c2c333" exitCode=60
Nov 02 18:00:08.511747 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.511411    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66" event={"ID":"3c1340c7-b300-40a2-a155-1f43448eccf4","Type":"ContainerDied","Data":"f92b5f0b99c3730fcaff1639751164dd2a42d2277b8537a3af06737225c2c333"}
Nov 02 18:00:08.512585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.512546    4456 scope.go:122] "RemoveContainer" containerID="f92b5f0b99c3730fcaff1639751164dd2a42d2277b8537a3af06737225c2c333"
Nov 02 18:00:08.513123 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:08.513102    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66_odh-ai-gateway-infra(3c1340c7-b300-40a2-a155-1f43448eccf4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66" podUID="3c1340c7-b300-40a2-a155-1f43448eccf4"
Nov 02 18:00:08.632775 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.632411    4456 scope.go:122] "RemoveContainer" containerID="bfdcc48b236f604ccfa345139725c90caec45907a37cc8499720ea67c4e0e2ae"
Nov 02 18:00:08.790875 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:08.790795    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf" podStartSLOduration=8.790775583 podStartE2EDuration="8.790775583s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:08.783560287 +0000 UTC m=+1825.274623450" watchObservedRunningTime="2026-11-02 18:00:08.790775583 +0000 UTC m=+1825.281838762"
Nov 02 18:00:09.527053 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.526898    4456 generic.go:356] "Generic (PLEG): container finished" podID="b21be16b-0021-4551-8662-a7c5fac4e7ad" containerID="75c8a99bc20c57716a347ea6decc0f470c63157109103a03bfd40d533447211c" exitCode=60
Nov 02 18:00:09.527053 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.527023    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf" event={"ID":"b21be16b-0021-4551-8662-a7c5fac4e7ad","Type":"ContainerDied","Data":"75c8a99bc20c57716a347ea6decc0f470c63157109103a03bfd40d533447211c"}
Nov 02 18:00:09.527904 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.527096    4456 scope.go:122] "RemoveContainer" containerID="e87e641d6e66ef09dde19101a44ef895c8587bd1e77ee2dd90cc0745f1dfdaeb"
Nov 02 18:00:09.527904 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.527693    4456 scope.go:122] "RemoveContainer" containerID="75c8a99bc20c57716a347ea6decc0f470c63157109103a03bfd40d533447211c"
Nov 02 18:00:09.528256 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:09.528223    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894040-nf8sf_odh-ai-gateway-infra(b21be16b-0021-4551-8662-a7c5fac4e7ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf" podUID="b21be16b-0021-4551-8662-a7c5fac4e7ad"
Nov 02 18:00:09.530437 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.529572    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc" event={"ID":"8765d0ad-655c-4f40-8825-5e0092d2c0aa","Type":"ContainerStarted","Data":"b97484deab90049bd19d8c54d2b1527eb229037f4b6c4488fef95e42ee927448"}
Nov 02 18:00:09.535128 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.535100    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9" event={"ID":"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211","Type":"ContainerStarted","Data":"48baf7d9aef5d803b521a6dae46e260fdc693970a4814fc87c56d812d0ca8bb4"}
Nov 02 18:00:09.537150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.537124    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw" event={"ID":"536e0993-02c4-43f0-a555-3bbadd26d1ab","Type":"ContainerStarted","Data":"60441804eb6be8d5a236bdd996047e8028ed804b97d99960f5d76a148d28a85a"}
Nov 02 18:00:09.547643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.547488    4456 generic.go:356] "Generic (PLEG): container finished" podID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab" containerID="d2b1d035332e9a95929d4ae6d5b0f6a6b0c668b0b33817b5fad7870e77591d23" exitCode=60
Nov 02 18:00:09.547643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.547579    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd" event={"ID":"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab","Type":"ContainerDied","Data":"d2b1d035332e9a95929d4ae6d5b0f6a6b0c668b0b33817b5fad7870e77591d23"}
Nov 02 18:00:09.548049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.548020    4456 scope.go:122] "RemoveContainer" containerID="d2b1d035332e9a95929d4ae6d5b0f6a6b0c668b0b33817b5fad7870e77591d23"
Nov 02 18:00:09.548653 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:09.548624    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd_odh-ai-gateway-infra(f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd" podUID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab"
Nov 02 18:00:09.550831 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.550795    4456 scope.go:122] "RemoveContainer" containerID="af88634edc19d3340f60eeabe5e9dc057f96b50fa23d222223800e3941efd065"
Nov 02 18:00:09.551194 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:09.551166    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b_odh-ai-gateway-infra(12b1615a-fc8e-4611-a7f7-84b25bd4295a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b" podUID="12b1615a-fc8e-4611-a7f7-84b25bd4295a"
Nov 02 18:00:09.552429 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.552400    4456 generic.go:356] "Generic (PLEG): container finished" podID="7e4a236e-0151-4496-a90d-58cccb957e91" containerID="f3a857086d513d872f21cdf2f22f5ca35db18f8fd0324054f7e804ca700576b6" exitCode=60
Nov 02 18:00:09.552534 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.552480    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z" event={"ID":"7e4a236e-0151-4496-a90d-58cccb957e91","Type":"ContainerDied","Data":"f3a857086d513d872f21cdf2f22f5ca35db18f8fd0324054f7e804ca700576b6"}
Nov 02 18:00:09.552693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.552677    4456 scope.go:122] "RemoveContainer" containerID="f3a857086d513d872f21cdf2f22f5ca35db18f8fd0324054f7e804ca700576b6"
Nov 02 18:00:09.553061 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:09.553034    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z_odh-ai-gateway-infra(7e4a236e-0151-4496-a90d-58cccb957e91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z" podUID="7e4a236e-0151-4496-a90d-58cccb957e91"
Nov 02 18:00:09.554585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.554559    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h" event={"ID":"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c","Type":"ContainerStarted","Data":"4a53762b69d110cd625f617fedf74ce3b3fcb2d7fe610ad9a598aa83eb973848"}
Nov 02 18:00:09.556922 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.556901    4456 scope.go:122] "RemoveContainer" containerID="00af0641b894f15ac2392bec7831e76c32daa4650b2e69d0afd1174f969c8554"
Nov 02 18:00:09.557236 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.557186    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h" event={"ID":"aba740c9-0c49-4641-83db-41bdbf8b9794","Type":"ContainerStarted","Data":"1611d0e04543c22b5ecefbb6a25350543b81c6580614f4ab66b03a0fca4c2442"}
Nov 02 18:00:09.559410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.559390    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc" event={"ID":"30b7824a-62b5-4395-8980-2d38452785a7","Type":"ContainerStarted","Data":"1a82d350791a4e172068efccfc8d66eea68a3734ac569b2755d457b74b92755d"}
Nov 02 18:00:09.561352 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.561333    4456 scope.go:122] "RemoveContainer" containerID="5f9a879d39b89e4236b9eb6225ce16584c956f7cd103cbc02ec62a4803cc1193"
Nov 02 18:00:09.561628 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:09.561608    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8_odh-ai-gateway-infra(59097f3d-9900-4383-9979-f3a60427b529)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8" podUID="59097f3d-9900-4383-9979-f3a60427b529"
Nov 02 18:00:09.563158 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.563132    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t" event={"ID":"661af531-d82f-434e-855d-516e8cc9caec","Type":"ContainerStarted","Data":"29b9a85e7897357b23d7ea32095d5c3fa046b0dbbde2f5d4d3c8549724895b1c"}
Nov 02 18:00:09.565016 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.564996    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5" event={"ID":"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4","Type":"ContainerStarted","Data":"2bf017fbed22826711fe9ae22711e08ded36f6875d492274aeedf4ff60a637ed"}
Nov 02 18:00:09.589354 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.589327    4456 scope.go:122] "RemoveContainer" containerID="4c3e7fe0716b1a0089dbad13de3e7165eac00a47f7fa46f97510d02003fdf244"
Nov 02 18:00:09.743723 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:09.743661    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5" podStartSLOduration=9.743645906 podStartE2EDuration="9.743645906s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:09.741804444 +0000 UTC m=+1826.232867624" watchObservedRunningTime="2026-11-02 18:00:09.743645906 +0000 UTC m=+1826.234709066"
Nov 02 18:00:10.042757 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.042676    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h" podStartSLOduration=10.042661814 podStartE2EDuration="10.042661814s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:10.039395464 +0000 UTC m=+1826.530458627" watchObservedRunningTime="2026-11-02 18:00:10.042661814 +0000 UTC m=+1826.533724976"
Nov 02 18:00:10.287840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.279430    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:10.585104 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.585015    4456 generic.go:356] "Generic (PLEG): container finished" podID="661af531-d82f-434e-855d-516e8cc9caec" containerID="29b9a85e7897357b23d7ea32095d5c3fa046b0dbbde2f5d4d3c8549724895b1c" exitCode=60
Nov 02 18:00:10.585104 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.585073    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t" event={"ID":"661af531-d82f-434e-855d-516e8cc9caec","Type":"ContainerDied","Data":"29b9a85e7897357b23d7ea32095d5c3fa046b0dbbde2f5d4d3c8549724895b1c"}
Nov 02 18:00:10.585620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.585203    4456 scope.go:122] "RemoveContainer" containerID="57846e6a62119a75db87ad23bc373079c09b2e8750f34250c2b93af806a6f358"
Nov 02 18:00:10.585681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.585670    4456 scope.go:122] "RemoveContainer" containerID="29b9a85e7897357b23d7ea32095d5c3fa046b0dbbde2f5d4d3c8549724895b1c"
Nov 02 18:00:10.586225 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:10.586185    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t_odh-ai-gateway-infra(661af531-d82f-434e-855d-516e8cc9caec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t" podUID="661af531-d82f-434e-855d-516e8cc9caec"
Nov 02 18:00:10.589375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.589351    4456 generic.go:356] "Generic (PLEG): container finished" podID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4" containerID="2bf017fbed22826711fe9ae22711e08ded36f6875d492274aeedf4ff60a637ed" exitCode=60
Nov 02 18:00:10.589514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.589424    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5" event={"ID":"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4","Type":"ContainerDied","Data":"2bf017fbed22826711fe9ae22711e08ded36f6875d492274aeedf4ff60a637ed"}
Nov 02 18:00:10.589886 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.589869    4456 scope.go:122] "RemoveContainer" containerID="2bf017fbed22826711fe9ae22711e08ded36f6875d492274aeedf4ff60a637ed"
Nov 02 18:00:10.590452 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:10.590418    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5_odh-ai-gateway-infra(40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5" podUID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4"
Nov 02 18:00:10.593426 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.593044    4456 scope.go:122] "RemoveContainer" containerID="75c8a99bc20c57716a347ea6decc0f470c63157109103a03bfd40d533447211c"
Nov 02 18:00:10.593625 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:10.593500    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894040-nf8sf_odh-ai-gateway-infra(b21be16b-0021-4551-8662-a7c5fac4e7ad)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf" podUID="b21be16b-0021-4551-8662-a7c5fac4e7ad"
Nov 02 18:00:10.596582 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.596540    4456 generic.go:356] "Generic (PLEG): container finished" podID="8765d0ad-655c-4f40-8825-5e0092d2c0aa" containerID="b97484deab90049bd19d8c54d2b1527eb229037f4b6c4488fef95e42ee927448" exitCode=60
Nov 02 18:00:10.596749 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.596724    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc" event={"ID":"8765d0ad-655c-4f40-8825-5e0092d2c0aa","Type":"ContainerDied","Data":"b97484deab90049bd19d8c54d2b1527eb229037f4b6c4488fef95e42ee927448"}
Nov 02 18:00:10.596984 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.596960    4456 scope.go:122] "RemoveContainer" containerID="b97484deab90049bd19d8c54d2b1527eb229037f4b6c4488fef95e42ee927448"
Nov 02 18:00:10.597411 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:10.597383    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc_odh-ai-gateway-infra(8765d0ad-655c-4f40-8825-5e0092d2c0aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc" podUID="8765d0ad-655c-4f40-8825-5e0092d2c0aa"
Nov 02 18:00:10.599381 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.599358    4456 generic.go:356] "Generic (PLEG): container finished" podID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211" containerID="48baf7d9aef5d803b521a6dae46e260fdc693970a4814fc87c56d812d0ca8bb4" exitCode=60
Nov 02 18:00:10.599485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.599451    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9" event={"ID":"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211","Type":"ContainerDied","Data":"48baf7d9aef5d803b521a6dae46e260fdc693970a4814fc87c56d812d0ca8bb4"}
Nov 02 18:00:10.599734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.599713    4456 scope.go:122] "RemoveContainer" containerID="48baf7d9aef5d803b521a6dae46e260fdc693970a4814fc87c56d812d0ca8bb4"
Nov 02 18:00:10.600229 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:10.600202    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9_odh-ai-gateway-infra(2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9" podUID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211"
Nov 02 18:00:10.603601 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.603578    4456 generic.go:356] "Generic (PLEG): container finished" podID="536e0993-02c4-43f0-a555-3bbadd26d1ab" containerID="60441804eb6be8d5a236bdd996047e8028ed804b97d99960f5d76a148d28a85a" exitCode=60
Nov 02 18:00:10.603742 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.603712    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw" event={"ID":"536e0993-02c4-43f0-a555-3bbadd26d1ab","Type":"ContainerDied","Data":"60441804eb6be8d5a236bdd996047e8028ed804b97d99960f5d76a148d28a85a"}
Nov 02 18:00:10.604149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.604030    4456 scope.go:122] "RemoveContainer" containerID="60441804eb6be8d5a236bdd996047e8028ed804b97d99960f5d76a148d28a85a"
Nov 02 18:00:10.604248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.604223    4456 scope.go:122] "RemoveContainer" containerID="10c7420291b19f55707bbc27977302ab54ccad3f97bdc7f028ada7e8737c5db0"
Nov 02 18:00:10.604407 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:10.604386    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw_odh-ai-gateway-infra(536e0993-02c4-43f0-a555-3bbadd26d1ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw" podUID="536e0993-02c4-43f0-a555-3bbadd26d1ab"
Nov 02 18:00:10.606858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.606838    4456 scope.go:122] "RemoveContainer" containerID="d2b1d035332e9a95929d4ae6d5b0f6a6b0c668b0b33817b5fad7870e77591d23"
Nov 02 18:00:10.607156 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:10.607135    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd_odh-ai-gateway-infra(f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd" podUID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab"
Nov 02 18:00:10.609459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.609440    4456 scope.go:122] "RemoveContainer" containerID="f3a857086d513d872f21cdf2f22f5ca35db18f8fd0324054f7e804ca700576b6"
Nov 02 18:00:10.609761 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:10.609737    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z_odh-ai-gateway-infra(7e4a236e-0151-4496-a90d-58cccb957e91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z" podUID="7e4a236e-0151-4496-a90d-58cccb957e91"
Nov 02 18:00:10.611472 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.611439    4456 generic.go:356] "Generic (PLEG): container finished" podID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" containerID="4a53762b69d110cd625f617fedf74ce3b3fcb2d7fe610ad9a598aa83eb973848" exitCode=60
Nov 02 18:00:10.611620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.611597    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h" event={"ID":"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c","Type":"ContainerDied","Data":"4a53762b69d110cd625f617fedf74ce3b3fcb2d7fe610ad9a598aa83eb973848"}
Nov 02 18:00:10.614867 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.614842    4456 scope.go:122] "RemoveContainer" containerID="4a53762b69d110cd625f617fedf74ce3b3fcb2d7fe610ad9a598aa83eb973848"
Nov 02 18:00:10.615232 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:10.615211    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h_odh-ai-gateway-infra(a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h" podUID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c"
Nov 02 18:00:10.617961 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.617935    4456 generic.go:356] "Generic (PLEG): container finished" podID="aba740c9-0c49-4641-83db-41bdbf8b9794" containerID="1611d0e04543c22b5ecefbb6a25350543b81c6580614f4ab66b03a0fca4c2442" exitCode=60
Nov 02 18:00:10.618066 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.618011    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h" event={"ID":"aba740c9-0c49-4641-83db-41bdbf8b9794","Type":"ContainerDied","Data":"1611d0e04543c22b5ecefbb6a25350543b81c6580614f4ab66b03a0fca4c2442"}
Nov 02 18:00:10.618275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.618256    4456 scope.go:122] "RemoveContainer" containerID="1611d0e04543c22b5ecefbb6a25350543b81c6580614f4ab66b03a0fca4c2442"
Nov 02 18:00:10.618589 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:10.618566    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h_odh-ai-gateway-infra(aba740c9-0c49-4641-83db-41bdbf8b9794)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h" podUID="aba740c9-0c49-4641-83db-41bdbf8b9794"
Nov 02 18:00:10.620251 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.620228    4456 generic.go:356] "Generic (PLEG): container finished" podID="30b7824a-62b5-4395-8980-2d38452785a7" containerID="1a82d350791a4e172068efccfc8d66eea68a3734ac569b2755d457b74b92755d" exitCode=60
Nov 02 18:00:10.620350 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.620265    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc" event={"ID":"30b7824a-62b5-4395-8980-2d38452785a7","Type":"ContainerDied","Data":"1a82d350791a4e172068efccfc8d66eea68a3734ac569b2755d457b74b92755d"}
Nov 02 18:00:10.620517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.620499    4456 scope.go:122] "RemoveContainer" containerID="1a82d350791a4e172068efccfc8d66eea68a3734ac569b2755d457b74b92755d"
Nov 02 18:00:10.620790 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:10.620768    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc_odh-ai-gateway-infra(30b7824a-62b5-4395-8980-2d38452785a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc" podUID="30b7824a-62b5-4395-8980-2d38452785a7"
Nov 02 18:00:10.626405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.626380    4456 scope.go:122] "RemoveContainer" containerID="46204cce816e9c8693ef3cff1a6e3f2a4a7281fa4db96873dff559d0333fb6f8"
Nov 02 18:00:10.649300 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.648474    4456 scope.go:122] "RemoveContainer" containerID="ffc33a8c2309405b44408086858e196af859d6ef8b95e2bdb1ccdd2bd32ec1d2"
Nov 02 18:00:10.671071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.671052    4456 scope.go:122] "RemoveContainer" containerID="52d18f2b9a2609d93fd4f7c5f0accfb026962f48c3f39d7464b1cf9cdc97d6d9"
Nov 02 18:00:10.688665 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.688644    4456 scope.go:122] "RemoveContainer" containerID="e5ceee86e2902811a1c2c933221cdbedfa5d44b83339c57136aacd58b08fb2b9"
Nov 02 18:00:10.703129 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.703110    4456 scope.go:122] "RemoveContainer" containerID="1557d8867d2ea3eeca999aed1b45377cf6be9b579cf51e8d96f47f7c485b90f8"
Nov 02 18:00:10.717480 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:10.717460    4456 scope.go:122] "RemoveContainer" containerID="54683bcc0e9c4b74a85a536b2d2a7355cc9decd07ff18ee37f8253e77f379c52"
Nov 02 18:00:11.632269 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:11.632235    4456 scope.go:122] "RemoveContainer" containerID="60441804eb6be8d5a236bdd996047e8028ed804b97d99960f5d76a148d28a85a"
Nov 02 18:00:11.632909 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:11.632705    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw_odh-ai-gateway-infra(536e0993-02c4-43f0-a555-3bbadd26d1ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw" podUID="536e0993-02c4-43f0-a555-3bbadd26d1ab"
Nov 02 18:00:11.634745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:11.634723    4456 scope.go:122] "RemoveContainer" containerID="4a53762b69d110cd625f617fedf74ce3b3fcb2d7fe610ad9a598aa83eb973848"
Nov 02 18:00:11.635147 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:11.635119    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h_odh-ai-gateway-infra(a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h" podUID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c"
Nov 02 18:00:11.636634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:11.636610    4456 scope.go:122] "RemoveContainer" containerID="1611d0e04543c22b5ecefbb6a25350543b81c6580614f4ab66b03a0fca4c2442"
Nov 02 18:00:11.637160 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:11.637133    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h_odh-ai-gateway-infra(aba740c9-0c49-4641-83db-41bdbf8b9794)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h" podUID="aba740c9-0c49-4641-83db-41bdbf8b9794"
Nov 02 18:00:11.638328 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:11.638299    4456 scope.go:122] "RemoveContainer" containerID="1a82d350791a4e172068efccfc8d66eea68a3734ac569b2755d457b74b92755d"
Nov 02 18:00:11.638611 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:11.638591    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc_odh-ai-gateway-infra(30b7824a-62b5-4395-8980-2d38452785a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc" podUID="30b7824a-62b5-4395-8980-2d38452785a7"
Nov 02 18:00:11.654183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:11.654164    4456 scope.go:122] "RemoveContainer" containerID="29b9a85e7897357b23d7ea32095d5c3fa046b0dbbde2f5d4d3c8549724895b1c"
Nov 02 18:00:11.654478 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:11.654458    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t_odh-ai-gateway-infra(661af531-d82f-434e-855d-516e8cc9caec)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t" podUID="661af531-d82f-434e-855d-516e8cc9caec"
Nov 02 18:00:11.656570 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:11.656554    4456 scope.go:122] "RemoveContainer" containerID="2bf017fbed22826711fe9ae22711e08ded36f6875d492274aeedf4ff60a637ed"
Nov 02 18:00:11.656880 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:11.656857    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5_odh-ai-gateway-infra(40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5" podUID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4"
Nov 02 18:00:11.658277 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:11.658259    4456 scope.go:122] "RemoveContainer" containerID="b97484deab90049bd19d8c54d2b1527eb229037f4b6c4488fef95e42ee927448"
Nov 02 18:00:11.658587 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:11.658558    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc_odh-ai-gateway-infra(8765d0ad-655c-4f40-8825-5e0092d2c0aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc" podUID="8765d0ad-655c-4f40-8825-5e0092d2c0aa"
Nov 02 18:00:11.660383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:11.660366    4456 scope.go:122] "RemoveContainer" containerID="48baf7d9aef5d803b521a6dae46e260fdc693970a4814fc87c56d812d0ca8bb4"
Nov 02 18:00:11.660615 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:11.660596    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9_odh-ai-gateway-infra(2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9" podUID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211"
Nov 02 18:00:12.183084 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:12.182956    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:13.962307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:13.962272    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:14.218376 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:14.218278    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:14.373186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:14.373141    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:15.719485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:15.719455    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:16.974342 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:16.862569    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:00:17.351531 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:17.351386    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:17.519839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:17.514885    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:19.214493 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:19.214435    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:19.401306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:19.401263    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:19.800316 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:19.800281    4456 scope.go:122] "RemoveContainer" containerID="f92b5f0b99c3730fcaff1639751164dd2a42d2277b8537a3af06737225c2c333"
Nov 02 18:00:20.557017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:20.556984    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"]
Nov 02 18:00:20.612750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:20.612470    4456 scope.go:122] "RemoveContainer" containerID="4a53762b69d110cd625f617fedf74ce3b3fcb2d7fe610ad9a598aa83eb973848"
Nov 02 18:00:20.639896 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:20.633670    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"]
Nov 02 18:00:20.802508 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:20.800382    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk" podUID="fe469a4c-b038-4869-afb1-5de7dfcebf93" containerName="istio-proxy" containerID="cri-o://0f1972bf9560563c35ab21bab295252d0e042c084612634d9a0d162bd1adb1e6" gracePeriod=30
Nov 02 18:00:21.018734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.018691    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"]
Nov 02 18:00:21.018940 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.018904    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.114158    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-podinfo\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.114196    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istiod-ca-cert\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.114322    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2vktq\" (UniqueName: \"kubernetes.io/projected/3c8d9150-66d1-4be8-8bca-3782fb0eac36-kube-api-access-2vktq\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.114467    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-workload-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.114588    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-envoy\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.114629    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-workload-certs\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.114736    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-credential-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.114760    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-data\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.114882    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-token\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.127985    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.217258    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-token\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.217402    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-podinfo\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.217443    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istiod-ca-cert\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.217501    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2vktq\" (UniqueName: \"kubernetes.io/projected/3c8d9150-66d1-4be8-8bca-3782fb0eac36-kube-api-access-2vktq\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.217682    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-workload-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.217943    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-envoy\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.217980    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-workload-certs\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.218225    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-credential-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.218254    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-data\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.218792    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-workload-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.218862    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-data\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.219143    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-workload-certs\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.219366    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-credential-socket\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.293874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.219378    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istiod-ca-cert\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.963760 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.607546    4456 scope.go:122] "RemoveContainer" containerID="75c8a99bc20c57716a347ea6decc0f470c63157109103a03bfd40d533447211c"
Nov 02 18:00:21.963760 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.608116    4456 scope.go:122] "RemoveContainer" containerID="2bf017fbed22826711fe9ae22711e08ded36f6875d492274aeedf4ff60a637ed"
Nov 02 18:00:21.963760 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.608382    4456 scope.go:122] "RemoveContainer" containerID="1a82d350791a4e172068efccfc8d66eea68a3734ac569b2755d457b74b92755d"
Nov 02 18:00:21.963760 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.936960    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-podinfo\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.963760 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.937110    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2vktq\" (UniqueName: \"kubernetes.io/projected/3c8d9150-66d1-4be8-8bca-3782fb0eac36-kube-api-access-2vktq\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.963760 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.937135    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-envoy\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:21.963760 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:21.937152    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-token\") pod \"e2e-inf-a-872354-openshift-default-59df9df7-vnmfl\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") " pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:22.333850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:22.171720    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66" event={"ID":"3c1340c7-b300-40a2-a155-1f43448eccf4","Type":"ContainerStarted","Data":"77865144e9748d83ee971d721bec11fd2bc258c464eaa1c0b0bc13e3887ffb59"}
Nov 02 18:00:22.333850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:22.173999    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h" event={"ID":"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c","Type":"ContainerStarted","Data":"be003e1d3789cf318bcc07a5108e64400abdbe4a7858213638a6543c6e686e09"}
Nov 02 18:00:22.333850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:22.187038    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:22.671234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:22.607093    4456 scope.go:122] "RemoveContainer" containerID="1611d0e04543c22b5ecefbb6a25350543b81c6580614f4ab66b03a0fca4c2442"
Nov 02 18:00:22.671234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:22.607332    4456 scope.go:122] "RemoveContainer" containerID="60441804eb6be8d5a236bdd996047e8028ed804b97d99960f5d76a148d28a85a"
Nov 02 18:00:22.671234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:22.607688    4456 scope.go:122] "RemoveContainer" containerID="f3a857086d513d872f21cdf2f22f5ca35db18f8fd0324054f7e804ca700576b6"
Nov 02 18:00:22.671234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:22.607830    4456 scope.go:122] "RemoveContainer" containerID="af88634edc19d3340f60eeabe5e9dc057f96b50fa23d222223800e3941efd065"
Nov 02 18:00:22.671234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:22.608069    4456 scope.go:122] "RemoveContainer" containerID="b97484deab90049bd19d8c54d2b1527eb229037f4b6c4488fef95e42ee927448"
Nov 02 18:00:22.671234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:22.608305    4456 scope.go:122] "RemoveContainer" containerID="5f9a879d39b89e4236b9eb6225ce16584c956f7cd103cbc02ec62a4803cc1193"
Nov 02 18:00:23.015684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:22.871595    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:23.226176 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:23.187403    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc" event={"ID":"30b7824a-62b5-4395-8980-2d38452785a7","Type":"ContainerStarted","Data":"dd9e42e86375f3bad6ff2ae34d0dc761943cbb908c2b21b43d7e0dca69af4c2c"}
Nov 02 18:00:23.226176 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:23.189450    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5" event={"ID":"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4","Type":"ContainerStarted","Data":"6df1455385922fb637a27294650f0f998b9a571aea315878ce800a693a561e62"}
Nov 02 18:00:23.248609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:23.228908    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"]
Nov 02 18:00:23.248609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:23.229070    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h" podUID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" containerName="cleanup" containerID="cri-o://be003e1d3789cf318bcc07a5108e64400abdbe4a7858213638a6543c6e686e09" gracePeriod=30
Nov 02 18:00:23.594802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:23.494966    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:00:23.594802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:23.495012    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:23.615530 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:23.615502    4456 scope.go:122] "RemoveContainer" containerID="d2b1d035332e9a95929d4ae6d5b0f6a6b0c668b0b33817b5fad7870e77591d23"
Nov 02 18:00:23.615693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:23.615609    4456 scope.go:122] "RemoveContainer" containerID="48baf7d9aef5d803b521a6dae46e260fdc693970a4814fc87c56d812d0ca8bb4"
Nov 02 18:00:24.247851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:24.200851    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf" event={"ID":"b21be16b-0021-4551-8662-a7c5fac4e7ad","Type":"ContainerStarted","Data":"b03068f57923f41a213e136272052bf6617f46bf2f8e3110fa00c525d74d1c8c"}
Nov 02 18:00:24.247851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:24.204696    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc" event={"ID":"8765d0ad-655c-4f40-8825-5e0092d2c0aa","Type":"ContainerStarted","Data":"74b8fb31a3dabe02bc7949d73cf64ab2bd3722fc1f8e4f69fb4835b63ce9562b"}
Nov 02 18:00:24.247851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:24.207279    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw" event={"ID":"536e0993-02c4-43f0-a555-3bbadd26d1ab","Type":"ContainerStarted","Data":"f9ae58eeb9f2a90edaa869a864546137f8aca1fd8686e84d6d7f793b0ba1d415"}
Nov 02 18:00:24.247851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:24.209806    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b" event={"ID":"12b1615a-fc8e-4611-a7f7-84b25bd4295a","Type":"ContainerStarted","Data":"e436c257d5fb80b719297e7d55aad59d2384a21280651cfa28acd02ac75024ce"}
Nov 02 18:00:24.247851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:24.212068    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z" event={"ID":"7e4a236e-0151-4496-a90d-58cccb957e91","Type":"ContainerStarted","Data":"1ee77135662ebda92c15308f539b08be83197cef1bb07ee09cb1dc7f7909c4e7"}
Nov 02 18:00:24.247851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:24.214379    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h" event={"ID":"aba740c9-0c49-4641-83db-41bdbf8b9794","Type":"ContainerStarted","Data":"4024178cecf3341a00a2fdeac99488aebb8e9fe5ef28ab9f9d1a7745391339cd"}
Nov 02 18:00:24.247851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:24.216671    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8" event={"ID":"59097f3d-9900-4383-9979-f3a60427b529","Type":"ContainerStarted","Data":"4c635f7ffb7816f532986250cf078690b433eb2eb5a2e61177392c0d48e9fcbd"}
Nov 02 18:00:24.247851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:24.245617    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"]
Nov 02 18:00:24.247851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:24.245859    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66" podUID="3c1340c7-b300-40a2-a155-1f43448eccf4" containerName="cleanup" containerID="cri-o://77865144e9748d83ee971d721bec11fd2bc258c464eaa1c0b0bc13e3887ffb59" gracePeriod=30
Nov 02 18:00:24.334414 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:24.289524    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8" podStartSLOduration=24.289506169 podStartE2EDuration="24.289506169s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:24.284487018 +0000 UTC m=+1840.775550172" watchObservedRunningTime="2026-11-02 18:00:24.289506169 +0000 UTC m=+1840.780569330"
Nov 02 18:00:24.458049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:24.458015    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:24.771889 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:24.609757    4456 scope.go:122] "RemoveContainer" containerID="29b9a85e7897357b23d7ea32095d5c3fa046b0dbbde2f5d4d3c8549724895b1c"
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.233514    4456 generic.go:356] "Generic (PLEG): container finished" podID="3c1340c7-b300-40a2-a155-1f43448eccf4" containerID="77865144e9748d83ee971d721bec11fd2bc258c464eaa1c0b0bc13e3887ffb59" exitCode=60
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.233585    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66" event={"ID":"3c1340c7-b300-40a2-a155-1f43448eccf4","Type":"ContainerDied","Data":"77865144e9748d83ee971d721bec11fd2bc258c464eaa1c0b0bc13e3887ffb59"}
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.233654    4456 scope.go:122] "RemoveContainer" containerID="f92b5f0b99c3730fcaff1639751164dd2a42d2277b8537a3af06737225c2c333"
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.237431    4456 generic.go:356] "Generic (PLEG): container finished" podID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211" containerID="10821d1d0cc7327011ba4b6f242711cc5d322f1606d9c5f417f1463dab984ed2" exitCode=60
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.237505    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9" event={"ID":"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211","Type":"ContainerDied","Data":"10821d1d0cc7327011ba4b6f242711cc5d322f1606d9c5f417f1463dab984ed2"}
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.237943    4456 scope.go:122] "RemoveContainer" containerID="10821d1d0cc7327011ba4b6f242711cc5d322f1606d9c5f417f1463dab984ed2"
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:25.238302    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9_odh-ai-gateway-infra(2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9" podUID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211"
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.239324    4456 generic.go:356] "Generic (PLEG): container finished" podID="536e0993-02c4-43f0-a555-3bbadd26d1ab" containerID="f9ae58eeb9f2a90edaa869a864546137f8aca1fd8686e84d6d7f793b0ba1d415" exitCode=60
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.239396    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw" event={"ID":"536e0993-02c4-43f0-a555-3bbadd26d1ab","Type":"ContainerDied","Data":"f9ae58eeb9f2a90edaa869a864546137f8aca1fd8686e84d6d7f793b0ba1d415"}
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.239714    4456 scope.go:122] "RemoveContainer" containerID="f9ae58eeb9f2a90edaa869a864546137f8aca1fd8686e84d6d7f793b0ba1d415"
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:25.240058    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw_odh-ai-gateway-infra(536e0993-02c4-43f0-a555-3bbadd26d1ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw" podUID="536e0993-02c4-43f0-a555-3bbadd26d1ab"
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.241189    4456 generic.go:356] "Generic (PLEG): container finished" podID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab" containerID="3abadee310d64b2386de5b944ee8234f7b8c70bbe5cc88bdc1221a56b057f7bd" exitCode=60
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.241243    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd" event={"ID":"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab","Type":"ContainerDied","Data":"3abadee310d64b2386de5b944ee8234f7b8c70bbe5cc88bdc1221a56b057f7bd"}
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.241538    4456 scope.go:122] "RemoveContainer" containerID="3abadee310d64b2386de5b944ee8234f7b8c70bbe5cc88bdc1221a56b057f7bd"
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:25.241831    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd_odh-ai-gateway-infra(f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd" podUID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab"
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.243099    4456 generic.go:356] "Generic (PLEG): container finished" podID="12b1615a-fc8e-4611-a7f7-84b25bd4295a" containerID="e436c257d5fb80b719297e7d55aad59d2384a21280651cfa28acd02ac75024ce" exitCode=60
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.243167    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b" event={"ID":"12b1615a-fc8e-4611-a7f7-84b25bd4295a","Type":"ContainerDied","Data":"e436c257d5fb80b719297e7d55aad59d2384a21280651cfa28acd02ac75024ce"}
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.243391    4456 scope.go:122] "RemoveContainer" containerID="e436c257d5fb80b719297e7d55aad59d2384a21280651cfa28acd02ac75024ce"
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:25.243657    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b_odh-ai-gateway-infra(12b1615a-fc8e-4611-a7f7-84b25bd4295a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b" podUID="12b1615a-fc8e-4611-a7f7-84b25bd4295a"
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.244862    4456 generic.go:356] "Generic (PLEG): container finished" podID="7e4a236e-0151-4496-a90d-58cccb957e91" containerID="1ee77135662ebda92c15308f539b08be83197cef1bb07ee09cb1dc7f7909c4e7" exitCode=60
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.244933    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z" event={"ID":"7e4a236e-0151-4496-a90d-58cccb957e91","Type":"ContainerDied","Data":"1ee77135662ebda92c15308f539b08be83197cef1bb07ee09cb1dc7f7909c4e7"}
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.245235    4456 scope.go:122] "RemoveContainer" containerID="1ee77135662ebda92c15308f539b08be83197cef1bb07ee09cb1dc7f7909c4e7"
Nov 02 18:00:25.246289 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:25.245489    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z_odh-ai-gateway-infra(7e4a236e-0151-4496-a90d-58cccb957e91)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z" podUID="7e4a236e-0151-4496-a90d-58cccb957e91"
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.246831    4456 generic.go:356] "Generic (PLEG): container finished" podID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" containerID="be003e1d3789cf318bcc07a5108e64400abdbe4a7858213638a6543c6e686e09" exitCode=60
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.246844    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h" event={"ID":"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c","Type":"ContainerDied","Data":"be003e1d3789cf318bcc07a5108e64400abdbe4a7858213638a6543c6e686e09"}
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.248717    4456 generic.go:356] "Generic (PLEG): container finished" podID="aba740c9-0c49-4641-83db-41bdbf8b9794" containerID="4024178cecf3341a00a2fdeac99488aebb8e9fe5ef28ab9f9d1a7745391339cd" exitCode=60
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.248796    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h" event={"ID":"aba740c9-0c49-4641-83db-41bdbf8b9794","Type":"ContainerDied","Data":"4024178cecf3341a00a2fdeac99488aebb8e9fe5ef28ab9f9d1a7745391339cd"}
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.249038    4456 scope.go:122] "RemoveContainer" containerID="4024178cecf3341a00a2fdeac99488aebb8e9fe5ef28ab9f9d1a7745391339cd"
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:25.249298    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h_odh-ai-gateway-infra(aba740c9-0c49-4641-83db-41bdbf8b9794)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h" podUID="aba740c9-0c49-4641-83db-41bdbf8b9794"
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.250763    4456 generic.go:356] "Generic (PLEG): container finished" podID="30b7824a-62b5-4395-8980-2d38452785a7" containerID="dd9e42e86375f3bad6ff2ae34d0dc761943cbb908c2b21b43d7e0dca69af4c2c" exitCode=60
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.250785    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc" event={"ID":"30b7824a-62b5-4395-8980-2d38452785a7","Type":"ContainerDied","Data":"dd9e42e86375f3bad6ff2ae34d0dc761943cbb908c2b21b43d7e0dca69af4c2c"}
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.251667    4456 scope.go:122] "RemoveContainer" containerID="dd9e42e86375f3bad6ff2ae34d0dc761943cbb908c2b21b43d7e0dca69af4c2c"
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:25.251935    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc_odh-ai-gateway-infra(30b7824a-62b5-4395-8980-2d38452785a7)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc" podUID="30b7824a-62b5-4395-8980-2d38452785a7"
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.253284    4456 generic.go:356] "Generic (PLEG): container finished" podID="59097f3d-9900-4383-9979-f3a60427b529" containerID="4c635f7ffb7816f532986250cf078690b433eb2eb5a2e61177392c0d48e9fcbd" exitCode=60
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.253354    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8" event={"ID":"59097f3d-9900-4383-9979-f3a60427b529","Type":"ContainerDied","Data":"4c635f7ffb7816f532986250cf078690b433eb2eb5a2e61177392c0d48e9fcbd"}
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.253634    4456 scope.go:122] "RemoveContainer" containerID="4c635f7ffb7816f532986250cf078690b433eb2eb5a2e61177392c0d48e9fcbd"
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:25.253942    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8_odh-ai-gateway-infra(59097f3d-9900-4383-9979-f3a60427b529)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8" podUID="59097f3d-9900-4383-9979-f3a60427b529"
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.257550    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"]
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.300402    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"]
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.300786    4456 generic.go:356] "Generic (PLEG): container finished" podID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4" containerID="6df1455385922fb637a27294650f0f998b9a571aea315878ce800a693a561e62" exitCode=60
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.300922    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5" event={"ID":"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4","Type":"ContainerDied","Data":"6df1455385922fb637a27294650f0f998b9a571aea315878ce800a693a561e62"}
Nov 02 18:00:25.301547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.301429    4456 scope.go:122] "RemoveContainer" containerID="6df1455385922fb637a27294650f0f998b9a571aea315878ce800a693a561e62"
Nov 02 18:00:25.528177 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:25.302008    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5_odh-ai-gateway-infra(40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5" podUID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4"
Nov 02 18:00:25.528177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.304466    4456 generic.go:356] "Generic (PLEG): container finished" podID="b21be16b-0021-4551-8662-a7c5fac4e7ad" containerID="b03068f57923f41a213e136272052bf6617f46bf2f8e3110fa00c525d74d1c8c" exitCode=60
Nov 02 18:00:25.528177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.304572    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf" event={"ID":"b21be16b-0021-4551-8662-a7c5fac4e7ad","Type":"ContainerDied","Data":"b03068f57923f41a213e136272052bf6617f46bf2f8e3110fa00c525d74d1c8c"}
Nov 02 18:00:25.528177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.306839    4456 generic.go:356] "Generic (PLEG): container finished" podID="8765d0ad-655c-4f40-8825-5e0092d2c0aa" containerID="74b8fb31a3dabe02bc7949d73cf64ab2bd3722fc1f8e4f69fb4835b63ce9562b" exitCode=60
Nov 02 18:00:25.528177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.306917    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc" event={"ID":"8765d0ad-655c-4f40-8825-5e0092d2c0aa","Type":"ContainerDied","Data":"74b8fb31a3dabe02bc7949d73cf64ab2bd3722fc1f8e4f69fb4835b63ce9562b"}
Nov 02 18:00:25.528177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.307304    4456 scope.go:122] "RemoveContainer" containerID="74b8fb31a3dabe02bc7949d73cf64ab2bd3722fc1f8e4f69fb4835b63ce9562b"
Nov 02 18:00:25.528177 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:25.307741    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc_odh-ai-gateway-infra(8765d0ad-655c-4f40-8825-5e0092d2c0aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc" podUID="8765d0ad-655c-4f40-8825-5e0092d2c0aa"
Nov 02 18:00:25.528177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.339279    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"]
Nov 02 18:00:25.528177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.386772    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"]
Nov 02 18:00:25.528177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.430428    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"]
Nov 02 18:00:25.528177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:25.472475    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"]
Nov 02 18:00:26.292119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:26.268524    4456 scope.go:122] "RemoveContainer" containerID="48baf7d9aef5d803b521a6dae46e260fdc693970a4814fc87c56d812d0ca8bb4"
Nov 02 18:00:26.400848 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:26.295162    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"]
Nov 02 18:00:26.400848 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:26.321523    4456 scope.go:122] "RemoveContainer" containerID="4024178cecf3341a00a2fdeac99488aebb8e9fe5ef28ab9f9d1a7745391339cd"
Nov 02 18:00:26.400848 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:26.321790    4456 scope.go:122] "RemoveContainer" containerID="e436c257d5fb80b719297e7d55aad59d2384a21280651cfa28acd02ac75024ce"
Nov 02 18:00:26.400848 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:26.321968    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h_odh-ai-gateway-infra(aba740c9-0c49-4641-83db-41bdbf8b9794)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h" podUID="aba740c9-0c49-4641-83db-41bdbf8b9794"
Nov 02 18:00:26.400848 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:26.322025    4456 scope.go:122] "RemoveContainer" containerID="74b8fb31a3dabe02bc7949d73cf64ab2bd3722fc1f8e4f69fb4835b63ce9562b"
Nov 02 18:00:26.400848 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:26.322230    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b_odh-ai-gateway-infra(12b1615a-fc8e-4611-a7f7-84b25bd4295a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b" podUID="12b1615a-fc8e-4611-a7f7-84b25bd4295a"
Nov 02 18:00:26.400848 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:26.322466    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc_odh-ai-gateway-infra(8765d0ad-655c-4f40-8825-5e0092d2c0aa)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc" podUID="8765d0ad-655c-4f40-8825-5e0092d2c0aa"
Nov 02 18:00:26.400848 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:26.344734    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:26.607207 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:26.471486    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"]
Nov 02 18:00:26.607207 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:26.495082    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"]
Nov 02 18:00:26.607207 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:26.505292    4456 scope.go:122] "RemoveContainer" containerID="60441804eb6be8d5a236bdd996047e8028ed804b97d99960f5d76a148d28a85a"
Nov 02 18:00:26.607207 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:26.567485    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"]
Nov 02 18:00:26.795292 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:26.785806    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"]
Nov 02 18:00:26.822726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:26.822698    4456 scope.go:122] "RemoveContainer" containerID="d2b1d035332e9a95929d4ae6d5b0f6a6b0c668b0b33817b5fad7870e77591d23"
Nov 02 18:00:27.193003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:27.192975    4456 scope.go:122] "RemoveContainer" containerID="af88634edc19d3340f60eeabe5e9dc057f96b50fa23d222223800e3941efd065"
Nov 02 18:00:27.332470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:27.332435    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t" event={"ID":"661af531-d82f-434e-855d-516e8cc9caec","Type":"ContainerStarted","Data":"1381eb797a6ab6cd97f09af7d539da08d8fba70796b32ca3bbf6e8016929cb06"}
Nov 02 18:00:27.400878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:27.334364    4456 generic.go:356] "Generic (PLEG): container finished" podID="fe469a4c-b038-4869-afb1-5de7dfcebf93" containerID="0f1972bf9560563c35ab21bab295252d0e042c084612634d9a0d162bd1adb1e6" exitCode=0
Nov 02 18:00:27.400878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:27.334440    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk" event={"ID":"fe469a4c-b038-4869-afb1-5de7dfcebf93","Type":"ContainerDied","Data":"0f1972bf9560563c35ab21bab295252d0e042c084612634d9a0d162bd1adb1e6"}
Nov 02 18:00:27.495560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:27.459656    4456 scope.go:122] "RemoveContainer" containerID="f3a857086d513d872f21cdf2f22f5ca35db18f8fd0324054f7e804ca700576b6"
Nov 02 18:00:27.726283 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:27.726251    4456 scope.go:122] "RemoveContainer" containerID="4a53762b69d110cd625f617fedf74ce3b3fcb2d7fe610ad9a598aa83eb973848"
Nov 02 18:00:27.875582 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:27.875555    4456 scope.go:122] "RemoveContainer" containerID="1611d0e04543c22b5ecefbb6a25350543b81c6580614f4ab66b03a0fca4c2442"
Nov 02 18:00:28.046011 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.045220    4456 scope.go:122] "RemoveContainer" containerID="1a82d350791a4e172068efccfc8d66eea68a3734ac569b2755d457b74b92755d"
Nov 02 18:00:28.175949 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.128739    4456 scope.go:122] "RemoveContainer" containerID="5f9a879d39b89e4236b9eb6225ce16584c956f7cd103cbc02ec62a4803cc1193"
Nov 02 18:00:28.175949 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.144050    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:28.280260 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.272290    4456 scope.go:122] "RemoveContainer" containerID="2bf017fbed22826711fe9ae22711e08ded36f6875d492274aeedf4ff60a637ed"
Nov 02 18:00:28.415558 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.415528    4456 scope.go:122] "RemoveContainer" containerID="75c8a99bc20c57716a347ea6decc0f470c63157109103a03bfd40d533447211c"
Nov 02 18:00:28.472349 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.441874    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"]
Nov 02 18:00:28.472349 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.454429    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused" start-of-body=
Nov 02 18:00:28.472349 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.454489    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk" podUID="fe469a4c-b038-4869-afb1-5de7dfcebf93" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.92:15021/healthz/ready\": dial tcp 10.134.0.92:15021: connect: connection refused"
Nov 02 18:00:28.899051 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.512921    4456 generic.go:356] "Generic (PLEG): container finished" podID="661af531-d82f-434e-855d-516e8cc9caec" containerID="1381eb797a6ab6cd97f09af7d539da08d8fba70796b32ca3bbf6e8016929cb06" exitCode=60
Nov 02 18:00:28.899051 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.513468    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t" event={"ID":"661af531-d82f-434e-855d-516e8cc9caec","Type":"ContainerDied","Data":"1381eb797a6ab6cd97f09af7d539da08d8fba70796b32ca3bbf6e8016929cb06"}
Nov 02 18:00:28.899051 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.610628    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:28.899051 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.636852    4456 scope.go:122] "RemoveContainer" containerID="b97484deab90049bd19d8c54d2b1527eb229037f4b6c4488fef95e42ee927448"
Nov 02 18:00:28.899051 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.831969    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:28.899051 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.872126    4456 scope.go:122] "RemoveContainer" containerID="29b9a85e7897357b23d7ea32095d5c3fa046b0dbbde2f5d4d3c8549724895b1c"
Nov 02 18:00:28.953437 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.906014    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"
Nov 02 18:00:28.953437 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.946353    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"
Nov 02 18:00:28.975244 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.961354    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"
Nov 02 18:00:28.986256 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.978226    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"
Nov 02 18:00:29.001547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.992581    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/59097f3d-9900-4383-9979-f3a60427b529-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59097f3d-9900-4383-9979-f3a60427b529-maas-api-service-ca\") pod \"59097f3d-9900-4383-9979-f3a60427b529\" (UID: \"59097f3d-9900-4383-9979-f3a60427b529\") "
Nov 02 18:00:29.001547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.992620    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/59097f3d-9900-4383-9979-f3a60427b529-kube-api-access-9c7fm\" (UniqueName: \"kubernetes.io/projected/59097f3d-9900-4383-9979-f3a60427b529-kube-api-access-9c7fm\") pod \"59097f3d-9900-4383-9979-f3a60427b529\" (UID: \"59097f3d-9900-4383-9979-f3a60427b529\") "
Nov 02 18:00:29.001547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.992714    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211-kube-api-access-znnn9\" (UniqueName: \"kubernetes.io/projected/2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211-kube-api-access-znnn9\") pod \"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211\" (UID: \"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211\") "
Nov 02 18:00:29.001547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.992800    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211-maas-api-service-ca\") pod \"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211\" (UID: \"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211\") "
Nov 02 18:00:29.001547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.992967    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8765d0ad-655c-4f40-8825-5e0092d2c0aa-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8765d0ad-655c-4f40-8825-5e0092d2c0aa-maas-api-service-ca\") pod \"8765d0ad-655c-4f40-8825-5e0092d2c0aa\" (UID: \"8765d0ad-655c-4f40-8825-5e0092d2c0aa\") "
Nov 02 18:00:29.001547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.993042    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8765d0ad-655c-4f40-8825-5e0092d2c0aa-kube-api-access-9dv4n\" (UniqueName: \"kubernetes.io/projected/8765d0ad-655c-4f40-8825-5e0092d2c0aa-kube-api-access-9dv4n\") pod \"8765d0ad-655c-4f40-8825-5e0092d2c0aa\" (UID: \"8765d0ad-655c-4f40-8825-5e0092d2c0aa\") "
Nov 02 18:00:29.001547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.993121    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c-kube-api-access-825ks\" (UniqueName: \"kubernetes.io/projected/a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c-kube-api-access-825ks\") pod \"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c\" (UID: \"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c\") "
Nov 02 18:00:29.001547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:28.993286    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c-maas-api-service-ca\") pod \"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c\" (UID: \"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c\") "
Nov 02 18:00:29.001547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.000127    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c-maas-api-service-ca" pod "a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" (UID: "a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:29.001547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.000572    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8765d0ad-655c-4f40-8825-5e0092d2c0aa-maas-api-service-ca" pod "8765d0ad-655c-4f40-8825-5e0092d2c0aa" (UID: "8765d0ad-655c-4f40-8825-5e0092d2c0aa"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:29.001547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.000754    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/59097f3d-9900-4383-9979-f3a60427b529-maas-api-service-ca" pod "59097f3d-9900-4383-9979-f3a60427b529" (UID: "59097f3d-9900-4383-9979-f3a60427b529"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:29.015955 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.002252    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211-maas-api-service-ca" pod "2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211" (UID: "2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:29.030181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.024582    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 18:00:29.068645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.052980    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.088742    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8765d0ad-655c-4f40-8825-5e0092d2c0aa-kube-api-access-9dv4n" pod "8765d0ad-655c-4f40-8825-5e0092d2c0aa" (UID: "8765d0ad-655c-4f40-8825-5e0092d2c0aa"). InnerVolumeSpecName "kube-api-access-9dv4n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.089354    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211-kube-api-access-znnn9" pod "2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211" (UID: "2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211"). InnerVolumeSpecName "kube-api-access-znnn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.089606    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/59097f3d-9900-4383-9979-f3a60427b529-kube-api-access-9c7fm" pod "59097f3d-9900-4383-9979-f3a60427b529" (UID: "59097f3d-9900-4383-9979-f3a60427b529"). InnerVolumeSpecName "kube-api-access-9c7fm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.089805    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c-kube-api-access-825ks" pod "a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" (UID: "a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c"). InnerVolumeSpecName "kube-api-access-825ks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.095009    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-workload-certs\") pod \"fe469a4c-b038-4869-afb1-5de7dfcebf93\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") "
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.095067    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e4a236e-0151-4496-a90d-58cccb957e91-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e4a236e-0151-4496-a90d-58cccb957e91-maas-api-service-ca\") pod \"7e4a236e-0151-4496-a90d-58cccb957e91\" (UID: \"7e4a236e-0151-4496-a90d-58cccb957e91\") "
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.095142    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-data\") pod \"fe469a4c-b038-4869-afb1-5de7dfcebf93\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") "
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.095216    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-workload-socket\") pod \"fe469a4c-b038-4869-afb1-5de7dfcebf93\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") "
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.095253    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-podinfo\") pod \"fe469a4c-b038-4869-afb1-5de7dfcebf93\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") "
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.095284    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-credential-socket\") pod \"fe469a4c-b038-4869-afb1-5de7dfcebf93\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") "
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.095362    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe469a4c-b038-4869-afb1-5de7dfcebf93-kube-api-access-rwklm\" (UniqueName: \"kubernetes.io/projected/fe469a4c-b038-4869-afb1-5de7dfcebf93-kube-api-access-rwklm\") pod \"fe469a4c-b038-4869-afb1-5de7dfcebf93\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") "
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.095522    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e4a236e-0151-4496-a90d-58cccb957e91-kube-api-access-k69sh\" (UniqueName: \"kubernetes.io/projected/7e4a236e-0151-4496-a90d-58cccb957e91-kube-api-access-k69sh\") pod \"7e4a236e-0151-4496-a90d-58cccb957e91\" (UID: \"7e4a236e-0151-4496-a90d-58cccb957e91\") "
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.095592    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fe469a4c-b038-4869-afb1-5de7dfcebf93-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe469a4c-b038-4869-afb1-5de7dfcebf93-istiod-ca-cert\") pod \"fe469a4c-b038-4869-afb1-5de7dfcebf93\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") "
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.095701    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-token\" (UniqueName: \"kubernetes.io/projected/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-token\") pod \"fe469a4c-b038-4869-afb1-5de7dfcebf93\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") "
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.095740    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-envoy\") pod \"fe469a4c-b038-4869-afb1-5de7dfcebf93\" (UID: \"fe469a4c-b038-4869-afb1-5de7dfcebf93\") "
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.096542    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-credential-socket" pod "fe469a4c-b038-4869-afb1-5de7dfcebf93" (UID: "fe469a4c-b038-4869-afb1-5de7dfcebf93"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.096548    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fe469a4c-b038-4869-afb1-5de7dfcebf93-istiod-ca-cert" pod "fe469a4c-b038-4869-afb1-5de7dfcebf93" (UID: "fe469a4c-b038-4869-afb1-5de7dfcebf93"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.096854    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-workload-certs" pod "fe469a4c-b038-4869-afb1-5de7dfcebf93" (UID: "fe469a4c-b038-4869-afb1-5de7dfcebf93"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.096879    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e4a236e-0151-4496-a90d-58cccb957e91-maas-api-service-ca" pod "7e4a236e-0151-4496-a90d-58cccb957e91" (UID: "7e4a236e-0151-4496-a90d-58cccb957e91"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.097089    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-workload-socket" pod "fe469a4c-b038-4869-afb1-5de7dfcebf93" (UID: "fe469a4c-b038-4869-afb1-5de7dfcebf93"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.097103    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-data" pod "fe469a4c-b038-4869-afb1-5de7dfcebf93" (UID: "fe469a4c-b038-4869-afb1-5de7dfcebf93"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.099973    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-znnn9\" (UniqueName: \"kubernetes.io/projected/2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211-kube-api-access-znnn9\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.099996    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.100054    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.100068    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/7e4a236e-0151-4496-a90d-58cccb957e91-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.100085    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/8765d0ad-655c-4f40-8825-5e0092d2c0aa-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.100098    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.100111    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9dv4n\" (UniqueName: \"kubernetes.io/projected/8765d0ad-655c-4f40-8825-5e0092d2c0aa-kube-api-access-9dv4n\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.100123    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.100136    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-825ks\" (UniqueName: \"kubernetes.io/projected/a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c-kube-api-access-825ks\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.100149    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.100163    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.100175    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/fe469a4c-b038-4869-afb1-5de7dfcebf93-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.100188    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/59097f3d-9900-4383-9979-f3a60427b529-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.100637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.100200    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9c7fm\" (UniqueName: \"kubernetes.io/projected/59097f3d-9900-4383-9979-f3a60427b529-kube-api-access-9c7fm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.107539 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.100670    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe469a4c-b038-4869-afb1-5de7dfcebf93-kube-api-access-rwklm" pod "fe469a4c-b038-4869-afb1-5de7dfcebf93" (UID: "fe469a4c-b038-4869-afb1-5de7dfcebf93"). InnerVolumeSpecName "kube-api-access-rwklm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.107539 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.101355    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-token" pod "fe469a4c-b038-4869-afb1-5de7dfcebf93" (UID: "fe469a4c-b038-4869-afb1-5de7dfcebf93"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.107539 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.106951    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-podinfo" pod "fe469a4c-b038-4869-afb1-5de7dfcebf93" (UID: "fe469a4c-b038-4869-afb1-5de7dfcebf93"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:00:29.107539 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.107011    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-envoy" pod "fe469a4c-b038-4869-afb1-5de7dfcebf93" (UID: "fe469a4c-b038-4869-afb1-5de7dfcebf93"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:29.154899 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.133947    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e4a236e-0151-4496-a90d-58cccb957e91-kube-api-access-k69sh" pod "7e4a236e-0151-4496-a90d-58cccb957e91" (UID: "7e4a236e-0151-4496-a90d-58cccb957e91"). InnerVolumeSpecName "kube-api-access-k69sh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.214612 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.198344    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"
Nov 02 18:00:29.214612 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.202435    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.214612 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.202465    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rwklm\" (UniqueName: \"kubernetes.io/projected/fe469a4c-b038-4869-afb1-5de7dfcebf93-kube-api-access-rwklm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.214612 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.202483    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k69sh\" (UniqueName: \"kubernetes.io/projected/7e4a236e-0151-4496-a90d-58cccb957e91-kube-api-access-k69sh\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.214612 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.202496    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.214612 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.202509    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/fe469a4c-b038-4869-afb1-5de7dfcebf93-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.214612 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.213020    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"
Nov 02 18:00:29.226589 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.221966    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"
Nov 02 18:00:29.233789 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.230976    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"
Nov 02 18:00:29.241982 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.240646    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"
Nov 02 18:00:29.255147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.255117    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"
Nov 02 18:00:29.264065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.264038    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"
Nov 02 18:00:29.270154 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.269323    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"
Nov 02 18:00:29.307210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.307141    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/536e0993-02c4-43f0-a555-3bbadd26d1ab-kube-api-access-ftrmx\" (UniqueName: \"kubernetes.io/projected/536e0993-02c4-43f0-a555-3bbadd26d1ab-kube-api-access-ftrmx\") pod \"536e0993-02c4-43f0-a555-3bbadd26d1ab\" (UID: \"536e0993-02c4-43f0-a555-3bbadd26d1ab\") "
Nov 02 18:00:29.307691 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.307625    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c1340c7-b300-40a2-a155-1f43448eccf4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1340c7-b300-40a2-a155-1f43448eccf4-maas-api-service-ca\") pod \"3c1340c7-b300-40a2-a155-1f43448eccf4\" (UID: \"3c1340c7-b300-40a2-a155-1f43448eccf4\") "
Nov 02 18:00:29.307691 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.307672    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aba740c9-0c49-4641-83db-41bdbf8b9794-kube-api-access-6kxls\" (UniqueName: \"kubernetes.io/projected/aba740c9-0c49-4641-83db-41bdbf8b9794-kube-api-access-6kxls\") pod \"aba740c9-0c49-4641-83db-41bdbf8b9794\" (UID: \"aba740c9-0c49-4641-83db-41bdbf8b9794\") "
Nov 02 18:00:29.308545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.307703    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aba740c9-0c49-4641-83db-41bdbf8b9794-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba740c9-0c49-4641-83db-41bdbf8b9794-maas-api-service-ca\") pod \"aba740c9-0c49-4641-83db-41bdbf8b9794\" (UID: \"aba740c9-0c49-4641-83db-41bdbf8b9794\") "
Nov 02 18:00:29.308545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.307832    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab-maas-api-service-ca\") pod \"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab\" (UID: \"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab\") "
Nov 02 18:00:29.308545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.307866    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30b7824a-62b5-4395-8980-2d38452785a7-kube-api-access-bm55m\" (UniqueName: \"kubernetes.io/projected/30b7824a-62b5-4395-8980-2d38452785a7-kube-api-access-bm55m\") pod \"30b7824a-62b5-4395-8980-2d38452785a7\" (UID: \"30b7824a-62b5-4395-8980-2d38452785a7\") "
Nov 02 18:00:29.308545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.307896    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b21be16b-0021-4551-8662-a7c5fac4e7ad-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b21be16b-0021-4551-8662-a7c5fac4e7ad-maas-api-service-ca\") pod \"b21be16b-0021-4551-8662-a7c5fac4e7ad\" (UID: \"b21be16b-0021-4551-8662-a7c5fac4e7ad\") "
Nov 02 18:00:29.308545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.307979    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab-kube-api-access-f5dtr\" (UniqueName: \"kubernetes.io/projected/f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab-kube-api-access-f5dtr\") pod \"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab\" (UID: \"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab\") "
Nov 02 18:00:29.308545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.308082    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c1340c7-b300-40a2-a155-1f43448eccf4-kube-api-access-jd5z4\" (UniqueName: \"kubernetes.io/projected/3c1340c7-b300-40a2-a155-1f43448eccf4-kube-api-access-jd5z4\") pod \"3c1340c7-b300-40a2-a155-1f43448eccf4\" (UID: \"3c1340c7-b300-40a2-a155-1f43448eccf4\") "
Nov 02 18:00:29.308545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.308113    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4-maas-api-service-ca\") pod \"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4\" (UID: \"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4\") "
Nov 02 18:00:29.308545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.308161    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/536e0993-02c4-43f0-a555-3bbadd26d1ab-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536e0993-02c4-43f0-a555-3bbadd26d1ab-maas-api-service-ca\") pod \"536e0993-02c4-43f0-a555-3bbadd26d1ab\" (UID: \"536e0993-02c4-43f0-a555-3bbadd26d1ab\") "
Nov 02 18:00:29.308545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.308204    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/12b1615a-fc8e-4611-a7f7-84b25bd4295a-kube-api-access-7d98v\" (UniqueName: \"kubernetes.io/projected/12b1615a-fc8e-4611-a7f7-84b25bd4295a-kube-api-access-7d98v\") pod \"12b1615a-fc8e-4611-a7f7-84b25bd4295a\" (UID: \"12b1615a-fc8e-4611-a7f7-84b25bd4295a\") "
Nov 02 18:00:29.308545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.308329    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30b7824a-62b5-4395-8980-2d38452785a7-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b7824a-62b5-4395-8980-2d38452785a7-maas-api-service-ca\") pod \"30b7824a-62b5-4395-8980-2d38452785a7\" (UID: \"30b7824a-62b5-4395-8980-2d38452785a7\") "
Nov 02 18:00:29.308545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.308379    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b21be16b-0021-4551-8662-a7c5fac4e7ad-kube-api-access-8kgmn\" (UniqueName: \"kubernetes.io/projected/b21be16b-0021-4551-8662-a7c5fac4e7ad-kube-api-access-8kgmn\") pod \"b21be16b-0021-4551-8662-a7c5fac4e7ad\" (UID: \"b21be16b-0021-4551-8662-a7c5fac4e7ad\") "
Nov 02 18:00:29.308545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.308477    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4-kube-api-access-m7qxx\" (UniqueName: \"kubernetes.io/projected/40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4-kube-api-access-m7qxx\") pod \"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4\" (UID: \"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4\") "
Nov 02 18:00:29.309195 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.308569    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/12b1615a-fc8e-4611-a7f7-84b25bd4295a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b1615a-fc8e-4611-a7f7-84b25bd4295a-maas-api-service-ca\") pod \"12b1615a-fc8e-4611-a7f7-84b25bd4295a\" (UID: \"12b1615a-fc8e-4611-a7f7-84b25bd4295a\") "
Nov 02 18:00:29.309586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.309519    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aba740c9-0c49-4641-83db-41bdbf8b9794-maas-api-service-ca" pod "aba740c9-0c49-4641-83db-41bdbf8b9794" (UID: "aba740c9-0c49-4641-83db-41bdbf8b9794"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:29.310011 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.309961    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c1340c7-b300-40a2-a155-1f43448eccf4-maas-api-service-ca" pod "3c1340c7-b300-40a2-a155-1f43448eccf4" (UID: "3c1340c7-b300-40a2-a155-1f43448eccf4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:29.311144 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.311115    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4-maas-api-service-ca" pod "40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4" (UID: "40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:29.311776 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.311750    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/12b1615a-fc8e-4611-a7f7-84b25bd4295a-maas-api-service-ca" pod "12b1615a-fc8e-4611-a7f7-84b25bd4295a" (UID: "12b1615a-fc8e-4611-a7f7-84b25bd4295a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:29.312126 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.312074    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/3c1340c7-b300-40a2-a155-1f43448eccf4-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.312126 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.312105    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/aba740c9-0c49-4641-83db-41bdbf8b9794-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.312126 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.312117    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.318015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.312129    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/12b1615a-fc8e-4611-a7f7-84b25bd4295a-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.318015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.312906    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/536e0993-02c4-43f0-a555-3bbadd26d1ab-maas-api-service-ca" pod "536e0993-02c4-43f0-a555-3bbadd26d1ab" (UID: "536e0993-02c4-43f0-a555-3bbadd26d1ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:29.318015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.313288    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab-maas-api-service-ca" pod "f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab" (UID: "f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:29.318015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.313667    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30b7824a-62b5-4395-8980-2d38452785a7-maas-api-service-ca" pod "30b7824a-62b5-4395-8980-2d38452785a7" (UID: "30b7824a-62b5-4395-8980-2d38452785a7"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:29.318015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.314002    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b21be16b-0021-4551-8662-a7c5fac4e7ad-maas-api-service-ca" pod "b21be16b-0021-4551-8662-a7c5fac4e7ad" (UID: "b21be16b-0021-4551-8662-a7c5fac4e7ad"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:29.318015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.317832    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/536e0993-02c4-43f0-a555-3bbadd26d1ab-kube-api-access-ftrmx" pod "536e0993-02c4-43f0-a555-3bbadd26d1ab" (UID: "536e0993-02c4-43f0-a555-3bbadd26d1ab"). InnerVolumeSpecName "kube-api-access-ftrmx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.326540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.326322    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30b7824a-62b5-4395-8980-2d38452785a7-kube-api-access-bm55m" pod "30b7824a-62b5-4395-8980-2d38452785a7" (UID: "30b7824a-62b5-4395-8980-2d38452785a7"). InnerVolumeSpecName "kube-api-access-bm55m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.332065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.332030    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4-kube-api-access-m7qxx" pod "40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4" (UID: "40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4"). InnerVolumeSpecName "kube-api-access-m7qxx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.333681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.333615    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/12b1615a-fc8e-4611-a7f7-84b25bd4295a-kube-api-access-7d98v" pod "12b1615a-fc8e-4611-a7f7-84b25bd4295a" (UID: "12b1615a-fc8e-4611-a7f7-84b25bd4295a"). InnerVolumeSpecName "kube-api-access-7d98v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.333681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.333654    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c1340c7-b300-40a2-a155-1f43448eccf4-kube-api-access-jd5z4" pod "3c1340c7-b300-40a2-a155-1f43448eccf4" (UID: "3c1340c7-b300-40a2-a155-1f43448eccf4"). InnerVolumeSpecName "kube-api-access-jd5z4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.335347 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.334985    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aba740c9-0c49-4641-83db-41bdbf8b9794-kube-api-access-6kxls" pod "aba740c9-0c49-4641-83db-41bdbf8b9794" (UID: "aba740c9-0c49-4641-83db-41bdbf8b9794"). InnerVolumeSpecName "kube-api-access-6kxls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.335347 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.335050    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab-kube-api-access-f5dtr" pod "f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab" (UID: "f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab"). InnerVolumeSpecName "kube-api-access-f5dtr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.335347 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.335161    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b21be16b-0021-4551-8662-a7c5fac4e7ad-kube-api-access-8kgmn" pod "b21be16b-0021-4551-8662-a7c5fac4e7ad" (UID: "b21be16b-0021-4551-8662-a7c5fac4e7ad"). InnerVolumeSpecName "kube-api-access-8kgmn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.421890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.401594    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"
Nov 02 18:00:29.421890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.413946    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/30b7824a-62b5-4395-8980-2d38452785a7-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.421890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.413976    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8kgmn\" (UniqueName: \"kubernetes.io/projected/b21be16b-0021-4551-8662-a7c5fac4e7ad-kube-api-access-8kgmn\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.421890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.413991    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-m7qxx\" (UniqueName: \"kubernetes.io/projected/40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4-kube-api-access-m7qxx\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.421890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.414005    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ftrmx\" (UniqueName: \"kubernetes.io/projected/536e0993-02c4-43f0-a555-3bbadd26d1ab-kube-api-access-ftrmx\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.421890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.414018    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6kxls\" (UniqueName: \"kubernetes.io/projected/aba740c9-0c49-4641-83db-41bdbf8b9794-kube-api-access-6kxls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.421890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.414032    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.421890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.414045    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bm55m\" (UniqueName: \"kubernetes.io/projected/30b7824a-62b5-4395-8980-2d38452785a7-kube-api-access-bm55m\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.421890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.414058    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/b21be16b-0021-4551-8662-a7c5fac4e7ad-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.421890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.414073    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5dtr\" (UniqueName: \"kubernetes.io/projected/f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab-kube-api-access-f5dtr\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.421890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.414085    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jd5z4\" (UniqueName: \"kubernetes.io/projected/3c1340c7-b300-40a2-a155-1f43448eccf4-kube-api-access-jd5z4\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.421890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.414098    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/536e0993-02c4-43f0-a555-3bbadd26d1ab-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.421890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.414111    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7d98v\" (UniqueName: \"kubernetes.io/projected/12b1615a-fc8e-4611-a7f7-84b25bd4295a-kube-api-access-7d98v\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.471832 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.471777    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:29.471970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.471890    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:29.471970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.471930    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:29.478264 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.478235    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"]
Nov 02 18:00:29.515762 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.515730    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/661af531-d82f-434e-855d-516e8cc9caec-kube-api-access-fvcjm\" (UniqueName: \"kubernetes.io/projected/661af531-d82f-434e-855d-516e8cc9caec-kube-api-access-fvcjm\") pod \"661af531-d82f-434e-855d-516e8cc9caec\" (UID: \"661af531-d82f-434e-855d-516e8cc9caec\") "
Nov 02 18:00:29.516201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.516177    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/661af531-d82f-434e-855d-516e8cc9caec-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/661af531-d82f-434e-855d-516e8cc9caec-maas-api-service-ca\") pod \"661af531-d82f-434e-855d-516e8cc9caec\" (UID: \"661af531-d82f-434e-855d-516e8cc9caec\") "
Nov 02 18:00:29.518094 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.518070    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/661af531-d82f-434e-855d-516e8cc9caec-maas-api-service-ca" pod "661af531-d82f-434e-855d-516e8cc9caec" (UID: "661af531-d82f-434e-855d-516e8cc9caec"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:29.521437 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.521416    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/661af531-d82f-434e-855d-516e8cc9caec-kube-api-access-fvcjm" pod "661af531-d82f-434e-855d-516e8cc9caec" (UID: "661af531-d82f-434e-855d-516e8cc9caec"). InnerVolumeSpecName "kube-api-access-fvcjm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:29.622432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.622370    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fvcjm\" (UniqueName: \"kubernetes.io/projected/661af531-d82f-434e-855d-516e8cc9caec-kube-api-access-fvcjm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.622432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.622402    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/661af531-d82f-434e-855d-516e8cc9caec-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:29.673120 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.673046    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h" event={"ID":"aba740c9-0c49-4641-83db-41bdbf8b9794","Type":"ContainerDied","Data":"230ceb62f1f8502bb24ee3a71890b4392571f9efdfee91fb07b88e7f7d940359"}
Nov 02 18:00:29.673334 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.673119    4456 scope.go:122] "RemoveContainer" containerID="4024178cecf3341a00a2fdeac99488aebb8e9fe5ef28ab9f9d1a7745391339cd"
Nov 02 18:00:29.676341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.675685    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"
Nov 02 18:00:29.683410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.683381    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc" event={"ID":"30b7824a-62b5-4395-8980-2d38452785a7","Type":"ContainerDied","Data":"f7d2b4f4666e90121df372558ff656b64536a5cd2acb020d7877969a2df95cc6"}
Nov 02 18:00:29.683541 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.683485    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"
Nov 02 18:00:29.689409 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.689380    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8" event={"ID":"59097f3d-9900-4383-9979-f3a60427b529","Type":"ContainerDied","Data":"1819ed936b96c80e788150ad0269c21e8586afae6451f67721bba2ff8f842cdd"}
Nov 02 18:00:29.689549 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.689477    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"
Nov 02 18:00:29.691772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.691718    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t" event={"ID":"661af531-d82f-434e-855d-516e8cc9caec","Type":"ContainerDied","Data":"185b56cb7b78f3b3062351903a83fe8d409c4b3c943dbfddc03f90b1b5c49136"}
Nov 02 18:00:29.691772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.691751    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"
Nov 02 18:00:29.693756 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.693731    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5" event={"ID":"40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4","Type":"ContainerDied","Data":"343b880cd19d87c1e057e7cf12151a7e9ba78b045e26d59e96f1c6e000e40a23"}
Nov 02 18:00:29.694209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.693808    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"
Nov 02 18:00:29.696917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.695529    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk" event={"ID":"fe469a4c-b038-4869-afb1-5de7dfcebf93","Type":"ContainerDied","Data":"90d1af4366439f70c41d98dd3af9389e9fadaded0a118451f7e8dbc681734725"}
Nov 02 18:00:29.696917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.695618    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"
Nov 02 18:00:29.715557 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.715530    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf" event={"ID":"b21be16b-0021-4551-8662-a7c5fac4e7ad","Type":"ContainerDied","Data":"b4c31a23f43cd47f4e4693ac8bbb9200dc93c44cb5f779d3eff934f36b5031db"}
Nov 02 18:00:29.715830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.715795    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"
Nov 02 18:00:29.718729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.718682    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" event={"ID":"3c8d9150-66d1-4be8-8bca-3782fb0eac36","Type":"ContainerStarted","Data":"db13028a213d83d2cc7de22049d0661e4b58c919befb6899b5eafb0bc7b2f632"}
Nov 02 18:00:29.778110 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.762266    4456 scope.go:122] "RemoveContainer" containerID="dd9e42e86375f3bad6ff2ae34d0dc761943cbb908c2b21b43d7e0dca69af4c2c"
Nov 02 18:00:29.833192 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.828712    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc" event={"ID":"8765d0ad-655c-4f40-8825-5e0092d2c0aa","Type":"ContainerDied","Data":"9f92b2913357352293c1e1a9e092c50277168f8b6a38adb51fa69ea492f6a8d4"}
Nov 02 18:00:29.833192 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.828965    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"
Nov 02 18:00:29.840496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.840433    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66" event={"ID":"3c1340c7-b300-40a2-a155-1f43448eccf4","Type":"ContainerDied","Data":"4b63477e8e5de8652303f9371bb54b74a35b29b5cef59ee69802402b8c145acb"}
Nov 02 18:00:29.840649 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.840573    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"
Nov 02 18:00:29.847050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.846377    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"
Nov 02 18:00:29.847233 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.847167    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9" event={"ID":"2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211","Type":"ContainerDied","Data":"cbf78392c72a42f1dd81e3ae5e1529bdeeff125fabcc90ae82dfaf6891cdca30"}
Nov 02 18:00:29.852958 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.852943    4456 scope.go:122] "RemoveContainer" containerID="4c635f7ffb7816f532986250cf078690b433eb2eb5a2e61177392c0d48e9fcbd"
Nov 02 18:00:29.868907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.854081    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"
Nov 02 18:00:29.868907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.854073    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw" event={"ID":"536e0993-02c4-43f0-a555-3bbadd26d1ab","Type":"ContainerDied","Data":"5175af497052a3a34468991569a3c6ea24bc3a0b51ccdedddb9c1b5c8a2c159e"}
Nov 02 18:00:29.868907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.855904    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd" event={"ID":"f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab","Type":"ContainerDied","Data":"ce601640105cce7c7cf188e745d7e0cdd0b89d7bbb3e15e6fa3f0fbb203f7161"}
Nov 02 18:00:29.868907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.856266    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"
Nov 02 18:00:29.868907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.857916    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b" event={"ID":"12b1615a-fc8e-4611-a7f7-84b25bd4295a","Type":"ContainerDied","Data":"d5750c3d3236c76e8788ab707585fcf1e15251331e22549c9c7862f049daf334"}
Nov 02 18:00:29.868907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.857998    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"
Nov 02 18:00:29.868907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.859896    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z" event={"ID":"7e4a236e-0151-4496-a90d-58cccb957e91","Type":"ContainerDied","Data":"69a10e840ec0ca789f01232cf891d5851370f482a3b5c78461158244fb89a74f"}
Nov 02 18:00:29.868907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.859910    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"
Nov 02 18:00:29.868907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.862366    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h" event={"ID":"a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c","Type":"ContainerDied","Data":"3be08b1c5df0cbd717c1d7a070a20235692993256920ea45b832148a5c4e1b2a"}
Nov 02 18:00:29.868907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.862440    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"
Nov 02 18:00:29.890916 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.890899    4456 scope.go:122] "RemoveContainer" containerID="1381eb797a6ab6cd97f09af7d539da08d8fba70796b32ca3bbf6e8016929cb06"
Nov 02 18:00:29.920210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.920179    4456 scope.go:122] "RemoveContainer" containerID="6df1455385922fb637a27294650f0f998b9a571aea315878ce800a693a561e62"
Nov 02 18:00:29.961564 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.961541    4456 scope.go:122] "RemoveContainer" containerID="0f1972bf9560563c35ab21bab295252d0e042c084612634d9a0d162bd1adb1e6"
Nov 02 18:00:29.986528 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.985441    4456 scope.go:122] "RemoveContainer" containerID="b03068f57923f41a213e136272052bf6617f46bf2f8e3110fa00c525d74d1c8c"
Nov 02 18:00:29.998954 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:29.998825    4456 scope.go:122] "RemoveContainer" containerID="74b8fb31a3dabe02bc7949d73cf64ab2bd3722fc1f8e4f69fb4835b63ce9562b"
Nov 02 18:00:30.023836 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.021227    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"]
Nov 02 18:00:30.034003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.032381    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-dxgpk"]
Nov 02 18:00:30.034003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.032478    4456 scope.go:122] "RemoveContainer" containerID="77865144e9748d83ee971d721bec11fd2bc258c464eaa1c0b0bc13e3887ffb59"
Nov 02 18:00:30.059672 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.058124    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"]
Nov 02 18:00:30.065233 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.065199    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w4-f59572-29894040-fv77t"]
Nov 02 18:00:30.066131 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.066035    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:30.092358 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.073175    4456 scope.go:122] "RemoveContainer" containerID="10821d1d0cc7327011ba4b6f242711cc5d322f1606d9c5f417f1463dab984ed2"
Nov 02 18:00:30.140748 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.140345    4456 scope.go:122] "RemoveContainer" containerID="f9ae58eeb9f2a90edaa869a864546137f8aca1fd8686e84d6d7f793b0ba1d415"
Nov 02 18:00:30.168986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.165982    4456 scope.go:122] "RemoveContainer" containerID="3abadee310d64b2386de5b944ee8234f7b8c70bbe5cc88bdc1221a56b057f7bd"
Nov 02 18:00:30.187807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.187749    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"]
Nov 02 18:00:30.193156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.193130    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894040-nf8sf"]
Nov 02 18:00:30.196241 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.196219    4456 scope.go:122] "RemoveContainer" containerID="e436c257d5fb80b719297e7d55aad59d2384a21280651cfa28acd02ac75024ce"
Nov 02 18:00:30.219489 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.218023    4456 scope.go:122] "RemoveContainer" containerID="1ee77135662ebda92c15308f539b08be83197cef1bb07ee09cb1dc7f7909c4e7"
Nov 02 18:00:30.234859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.234788    4456 scope.go:122] "RemoveContainer" containerID="be003e1d3789cf318bcc07a5108e64400abdbe4a7858213638a6543c6e686e09"
Nov 02 18:00:30.304719 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.304686    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"]
Nov 02 18:00:30.309681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.309652    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-bb6d2e-29894040-6st6h"]
Nov 02 18:00:30.347480 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.347450    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"]
Nov 02 18:00:30.352402 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.352378    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w5-863b87-29894040-wxfpc"]
Nov 02 18:00:30.389795 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.389764    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"]
Nov 02 18:00:30.395120 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.395096    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w3-5eedab-29894040-mzncc"]
Nov 02 18:00:30.443332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.443296    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"]
Nov 02 18:00:30.446335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.446293    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w0-f91b2d-29894040-7mw66"]
Nov 02 18:00:30.477737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.477705    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"]
Nov 02 18:00:30.482487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.482464    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894040-nm5wd"]
Nov 02 18:00:30.521015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.520957    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"]
Nov 02 18:00:30.530305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.530166    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-063f48-29894040-tsn7h"]
Nov 02 18:00:30.568321 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.568295    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"]
Nov 02 18:00:30.576589 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.576556    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-a-w5-504de7-29894040-vfjc8"]
Nov 02 18:00:30.604785 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.604580    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"]
Nov 02 18:00:30.618296 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.618261    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w1-120394-29894040-kbcnw"]
Nov 02 18:00:30.645084 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.645023    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"]
Nov 02 18:00:30.653163 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.653131    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w2-f0a3b8-29894040-8pwg9"]
Nov 02 18:00:30.703914 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.703794    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"]
Nov 02 18:00:30.709540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.709514    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-inf-a-872354-29894040-twnb5"]
Nov 02 18:00:30.740351 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.740321    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"]
Nov 02 18:00:30.745408 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.745381    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w5-c810ca-29894040-4sm8b"]
Nov 02 18:00:30.783783 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.783755    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"]
Nov 02 18:00:30.792638 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.792047    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w6-f6da5f-29894040-dl86z"]
Nov 02 18:00:30.902455 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.902415    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" event={"ID":"3c8d9150-66d1-4be8-8bca-3782fb0eac36","Type":"ContainerStarted","Data":"7b5ffd17510b581a401cae0b3017c71076c1341b3ae93eed0e79ed9040740ec5"}
Nov 02 18:00:30.955621 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:30.955521    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" podStartSLOduration=10.955505995 podStartE2EDuration="10.955505995s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:30.952716195 +0000 UTC m=+1847.443779369" watchObservedRunningTime="2026-11-02 18:00:30.955505995 +0000 UTC m=+1847.446569231"
Nov 02 18:00:31.187287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.187249    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:31.188919 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.188883    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 02 18:00:31.189082 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.188961    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 02 18:00:31.297868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.297765    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"]
Nov 02 18:00:31.299187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299159    4456 cpu_manager.go:414] "containerMap: removing container" podUID="59097f3d-9900-4383-9979-f3a60427b529" containerName="cleanup"
Nov 02 18:00:31.299395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299379    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59097f3d-9900-4383-9979-f3a60427b529" containerName="cleanup"
Nov 02 18:00:31.299497 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299486    4456 cpu_manager.go:414] "containerMap: removing container" podUID="b21be16b-0021-4551-8662-a7c5fac4e7ad" containerName="cleanup"
Nov 02 18:00:31.299575 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299565    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21be16b-0021-4551-8662-a7c5fac4e7ad" containerName="cleanup"
Nov 02 18:00:31.299667 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299658    4456 cpu_manager.go:414] "containerMap: removing container" podUID="536e0993-02c4-43f0-a555-3bbadd26d1ab" containerName="cleanup"
Nov 02 18:00:31.299741 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299731    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536e0993-02c4-43f0-a555-3bbadd26d1ab" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299854    4456 cpu_manager.go:414] "containerMap: removing container" podUID="aba740c9-0c49-4641-83db-41bdbf8b9794" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299865    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba740c9-0c49-4641-83db-41bdbf8b9794" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299879    4456 cpu_manager.go:414] "containerMap: removing container" podUID="30b7824a-62b5-4395-8980-2d38452785a7" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299888    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b7824a-62b5-4395-8980-2d38452785a7" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299902    4456 cpu_manager.go:414] "containerMap: removing container" podUID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299909    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299922    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299930    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299946    4456 cpu_manager.go:414] "containerMap: removing container" podUID="8765d0ad-655c-4f40-8825-5e0092d2c0aa" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299970    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8765d0ad-655c-4f40-8825-5e0092d2c0aa" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299986    4456 cpu_manager.go:414] "containerMap: removing container" podUID="3c1340c7-b300-40a2-a155-1f43448eccf4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.299993    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1340c7-b300-40a2-a155-1f43448eccf4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300004    4456 cpu_manager.go:414] "containerMap: removing container" podUID="536e0993-02c4-43f0-a555-3bbadd26d1ab" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300011    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536e0993-02c4-43f0-a555-3bbadd26d1ab" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300022    4456 cpu_manager.go:414] "containerMap: removing container" podUID="12b1615a-fc8e-4611-a7f7-84b25bd4295a" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300029    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b1615a-fc8e-4611-a7f7-84b25bd4295a" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300041    4456 cpu_manager.go:414] "containerMap: removing container" podUID="fe469a4c-b038-4869-afb1-5de7dfcebf93" containerName="istio-proxy"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300049    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fe469a4c-b038-4869-afb1-5de7dfcebf93" containerName="istio-proxy"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300071    4456 cpu_manager.go:414] "containerMap: removing container" podUID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300077    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300086    4456 cpu_manager.go:414] "containerMap: removing container" podUID="30b7824a-62b5-4395-8980-2d38452785a7" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300093    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b7824a-62b5-4395-8980-2d38452785a7" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300105    4456 cpu_manager.go:414] "containerMap: removing container" podUID="aba740c9-0c49-4641-83db-41bdbf8b9794" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300112    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba740c9-0c49-4641-83db-41bdbf8b9794" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300133    4456 cpu_manager.go:414] "containerMap: removing container" podUID="8765d0ad-655c-4f40-8825-5e0092d2c0aa" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300140    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8765d0ad-655c-4f40-8825-5e0092d2c0aa" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300152    4456 cpu_manager.go:414] "containerMap: removing container" podUID="30b7824a-62b5-4395-8980-2d38452785a7" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300158    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30b7824a-62b5-4395-8980-2d38452785a7" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300169    4456 cpu_manager.go:414] "containerMap: removing container" podUID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300175    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300189    4456 cpu_manager.go:414] "containerMap: removing container" podUID="7e4a236e-0151-4496-a90d-58cccb957e91" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300196    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4a236e-0151-4496-a90d-58cccb957e91" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300207    4456 cpu_manager.go:414] "containerMap: removing container" podUID="661af531-d82f-434e-855d-516e8cc9caec" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300214    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661af531-d82f-434e-855d-516e8cc9caec" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300231    4456 cpu_manager.go:414] "containerMap: removing container" podUID="3c1340c7-b300-40a2-a155-1f43448eccf4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300237    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1340c7-b300-40a2-a155-1f43448eccf4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300251    4456 cpu_manager.go:414] "containerMap: removing container" podUID="12b1615a-fc8e-4611-a7f7-84b25bd4295a" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300258    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b1615a-fc8e-4611-a7f7-84b25bd4295a" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300270    4456 cpu_manager.go:414] "containerMap: removing container" podUID="b21be16b-0021-4551-8662-a7c5fac4e7ad" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300276    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21be16b-0021-4551-8662-a7c5fac4e7ad" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300287    4456 cpu_manager.go:414] "containerMap: removing container" podUID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300294    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300308    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300315    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300332    4456 cpu_manager.go:414] "containerMap: removing container" podUID="aba740c9-0c49-4641-83db-41bdbf8b9794" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300338    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aba740c9-0c49-4641-83db-41bdbf8b9794" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300359    4456 cpu_manager.go:414] "containerMap: removing container" podUID="536e0993-02c4-43f0-a555-3bbadd26d1ab" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300366    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="536e0993-02c4-43f0-a555-3bbadd26d1ab" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300382    4456 cpu_manager.go:414] "containerMap: removing container" podUID="12b1615a-fc8e-4611-a7f7-84b25bd4295a" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300388    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="12b1615a-fc8e-4611-a7f7-84b25bd4295a" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300404    4456 cpu_manager.go:414] "containerMap: removing container" podUID="59097f3d-9900-4383-9979-f3a60427b529" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300430    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59097f3d-9900-4383-9979-f3a60427b529" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300445    4456 cpu_manager.go:414] "containerMap: removing container" podUID="661af531-d82f-434e-855d-516e8cc9caec" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300453    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661af531-d82f-434e-855d-516e8cc9caec" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300463    4456 cpu_manager.go:414] "containerMap: removing container" podUID="3c1340c7-b300-40a2-a155-1f43448eccf4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300470    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c1340c7-b300-40a2-a155-1f43448eccf4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300491    4456 cpu_manager.go:414] "containerMap: removing container" podUID="b21be16b-0021-4551-8662-a7c5fac4e7ad" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300500    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b21be16b-0021-4551-8662-a7c5fac4e7ad" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300515    4456 cpu_manager.go:414] "containerMap: removing container" podUID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300522    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300538    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300546    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300555    4456 cpu_manager.go:414] "containerMap: removing container" podUID="7e4a236e-0151-4496-a90d-58cccb957e91" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300562    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4a236e-0151-4496-a90d-58cccb957e91" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300573    4456 cpu_manager.go:414] "containerMap: removing container" podUID="8765d0ad-655c-4f40-8825-5e0092d2c0aa" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300580    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8765d0ad-655c-4f40-8825-5e0092d2c0aa" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300595    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300602    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300613    4456 cpu_manager.go:414] "containerMap: removing container" podUID="7e4a236e-0151-4496-a90d-58cccb957e91" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300620    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e4a236e-0151-4496-a90d-58cccb957e91" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300870    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b1615a-fc8e-4611-a7f7-84b25bd4295a" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300887    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300897    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="59097f3d-9900-4383-9979-f3a60427b529" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300909    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="661af531-d82f-434e-855d-516e8cc9caec" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300919    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1340c7-b300-40a2-a155-1f43448eccf4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300926    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300940    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21be16b-0021-4551-8662-a7c5fac4e7ad" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300948    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="661af531-d82f-434e-855d-516e8cc9caec" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300955    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300965    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300972    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4a236e-0151-4496-a90d-58cccb957e91" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300982    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8765d0ad-655c-4f40-8825-5e0092d2c0aa" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.300990    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4a236e-0151-4496-a90d-58cccb957e91" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301001    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="59097f3d-9900-4383-9979-f3a60427b529" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301013    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="536e0993-02c4-43f0-a555-3bbadd26d1ab" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301020    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba740c9-0c49-4641-83db-41bdbf8b9794" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301029    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b7824a-62b5-4395-8980-2d38452785a7" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301039    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301049    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1340c7-b300-40a2-a155-1f43448eccf4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301056    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="536e0993-02c4-43f0-a555-3bbadd26d1ab" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301063    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="fe469a4c-b038-4869-afb1-5de7dfcebf93" containerName="istio-proxy"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301073    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301084    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b7824a-62b5-4395-8980-2d38452785a7" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301153    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8765d0ad-655c-4f40-8825-5e0092d2c0aa" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301166    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="30b7824a-62b5-4395-8980-2d38452785a7" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301221    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b1615a-fc8e-4611-a7f7-84b25bd4295a" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301231    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21be16b-0021-4551-8662-a7c5fac4e7ad" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301241    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301253    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba740c9-0c49-4641-83db-41bdbf8b9794" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301262    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="536e0993-02c4-43f0-a555-3bbadd26d1ab" containerName="cleanup"
Nov 02 18:00:31.321137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.301270    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab" containerName="cleanup"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.371703    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"]
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.371858    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.379627    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-data\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.379688    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92ad0220-435f-4ae8-b743-3421ce98e6fb-istiod-ca-cert\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.379900    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-workload-certs\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.379960    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-workload-socket\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.379987    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-credential-socket\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.380054    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rrvm\" (UniqueName: \"kubernetes.io/projected/92ad0220-435f-4ae8-b743-3421ce98e6fb-kube-api-access-2rrvm\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.380106    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-podinfo\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.380158    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-token\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.380423    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-envoy\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.413184    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b2fc9f-openshift-default-dockercfg-tqz7r\""
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.512254    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92ad0220-435f-4ae8-b743-3421ce98e6fb-istiod-ca-cert\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.512376    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-workload-certs\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.512430    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-workload-socket\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.512460    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-credential-socket\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.512536    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rrvm\" (UniqueName: \"kubernetes.io/projected/92ad0220-435f-4ae8-b743-3421ce98e6fb-kube-api-access-2rrvm\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.512597    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-podinfo\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.512655    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-token\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.512946    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-envoy\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.513029    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-data\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.513471    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-credential-socket\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.513489    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-data\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.514138    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92ad0220-435f-4ae8-b743-3421ce98e6fb-istiod-ca-cert\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.514401    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-workload-socket\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.516998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.514577    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-workload-certs\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.519582    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"]
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520626    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520642    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520669    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520675    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520692    4456 cpu_manager.go:414] "containerMap: removing container" podUID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520697    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520733    4456 cpu_manager.go:414] "containerMap: removing container" podUID="59097f3d-9900-4383-9979-f3a60427b529" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520738    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="59097f3d-9900-4383-9979-f3a60427b529" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520745    4456 cpu_manager.go:414] "containerMap: removing container" podUID="661af531-d82f-434e-855d-516e8cc9caec" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520750    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="661af531-d82f-434e-855d-516e8cc9caec" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520903    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="12b1615a-fc8e-4611-a7f7-84b25bd4295a" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520913    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c1340c7-b300-40a2-a155-1f43448eccf4" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520919    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520930    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b21be16b-0021-4551-8662-a7c5fac4e7ad" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520937    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="661af531-d82f-434e-855d-516e8cc9caec" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520942    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520948    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520954    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e4a236e-0151-4496-a90d-58cccb957e91" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520960    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="59097f3d-9900-4383-9979-f3a60427b529" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520966    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="aba740c9-0c49-4641-83db-41bdbf8b9794" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520974    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.520982    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8765d0ad-655c-4f40-8825-5e0092d2c0aa" containerName="cleanup"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.536626    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-podinfo\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.538215    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-token\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.539017    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rrvm\" (UniqueName: \"kubernetes.io/projected/92ad0220-435f-4ae8-b743-3421ce98e6fb-kube-api-access-2rrvm\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.570147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.539166    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-envoy\") pod \"e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") " pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:31.579588 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.578941    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"]
Nov 02 18:00:31.579588 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.579030    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"
Nov 02 18:00:31.627278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.613511    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:00:31.627278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.613588    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:00:31.627278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.615731    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c504605b-6fqhs\" (UID: \"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"
Nov 02 18:00:31.627278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.626603    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5vqf9\" (UniqueName: \"kubernetes.io/projected/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-kube-api-access-5vqf9\") pod \"maas-api-revoke-sub-c504605b-6fqhs\" (UID: \"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"
Nov 02 18:00:31.627278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.627114    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c504605b-6fqhs\" (UID: \"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"
Nov 02 18:00:31.640416 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.640384    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="12b1615a-fc8e-4611-a7f7-84b25bd4295a" path="/var/lib/kubelet/pods/12b1615a-fc8e-4611-a7f7-84b25bd4295a/volumes"
Nov 02 18:00:31.641027 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.641007    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211" path="/var/lib/kubelet/pods/2f44ddc9-0014-4dbc-b6d5-0fb8e9db7211/volumes"
Nov 02 18:00:31.641529 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.641510    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30b7824a-62b5-4395-8980-2d38452785a7" path="/var/lib/kubelet/pods/30b7824a-62b5-4395-8980-2d38452785a7/volumes"
Nov 02 18:00:31.642342 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.642321    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c1340c7-b300-40a2-a155-1f43448eccf4" path="/var/lib/kubelet/pods/3c1340c7-b300-40a2-a155-1f43448eccf4/volumes"
Nov 02 18:00:31.642797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.642777    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4" path="/var/lib/kubelet/pods/40181b2b-2dd4-4f0b-bbcd-eccdc2a4c9b4/volumes"
Nov 02 18:00:31.643297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.643279    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="536e0993-02c4-43f0-a555-3bbadd26d1ab" path="/var/lib/kubelet/pods/536e0993-02c4-43f0-a555-3bbadd26d1ab/volumes"
Nov 02 18:00:31.644042 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.644021    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="59097f3d-9900-4383-9979-f3a60427b529" path="/var/lib/kubelet/pods/59097f3d-9900-4383-9979-f3a60427b529/volumes"
Nov 02 18:00:31.644501 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.644482    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="661af531-d82f-434e-855d-516e8cc9caec" path="/var/lib/kubelet/pods/661af531-d82f-434e-855d-516e8cc9caec/volumes"
Nov 02 18:00:31.644970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.644951    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e4a236e-0151-4496-a90d-58cccb957e91" path="/var/lib/kubelet/pods/7e4a236e-0151-4496-a90d-58cccb957e91/volumes"
Nov 02 18:00:31.645707 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.645682    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8765d0ad-655c-4f40-8825-5e0092d2c0aa" path="/var/lib/kubelet/pods/8765d0ad-655c-4f40-8825-5e0092d2c0aa/volumes"
Nov 02 18:00:31.646136 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.646121    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c" path="/var/lib/kubelet/pods/a3f246a3-64d1-422a-9e1e-8f42c8cfdd5c/volumes"
Nov 02 18:00:31.646532 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.646517    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aba740c9-0c49-4641-83db-41bdbf8b9794" path="/var/lib/kubelet/pods/aba740c9-0c49-4641-83db-41bdbf8b9794/volumes"
Nov 02 18:00:31.647442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.647422    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b21be16b-0021-4551-8662-a7c5fac4e7ad" path="/var/lib/kubelet/pods/b21be16b-0021-4551-8662-a7c5fac4e7ad/volumes"
Nov 02 18:00:31.647950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.647927    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab" path="/var/lib/kubelet/pods/f0d5c89a-7ef7-4103-8b1d-32f60ee3d3ab/volumes"
Nov 02 18:00:31.649130 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.649110    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fe469a4c-b038-4869-afb1-5de7dfcebf93" path="/var/lib/kubelet/pods/fe469a4c-b038-4869-afb1-5de7dfcebf93/volumes"
Nov 02 18:00:31.728608 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.728575    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5vqf9\" (UniqueName: \"kubernetes.io/projected/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-kube-api-access-5vqf9\") pod \"maas-api-revoke-sub-c504605b-6fqhs\" (UID: \"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"
Nov 02 18:00:31.728951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.728924    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c504605b-6fqhs\" (UID: \"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"
Nov 02 18:00:31.729087 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.729070    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c504605b-6fqhs\" (UID: \"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"
Nov 02 18:00:31.729650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.729627    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-c504605b-6fqhs\" (UID: \"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"
Nov 02 18:00:31.732221 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.732193    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-maas-api-xks-ca\") pod \"maas-api-revoke-sub-c504605b-6fqhs\" (UID: \"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"
Nov 02 18:00:31.739895 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.739869    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:31.789469 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.789258    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5vqf9\" (UniqueName: \"kubernetes.io/projected/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-kube-api-access-5vqf9\") pod \"maas-api-revoke-sub-c504605b-6fqhs\" (UID: \"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"
Nov 02 18:00:31.820584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:31.820175    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:32.043618 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.033016    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"
Nov 02 18:00:32.221843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.210668    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 02 18:00:32.221843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.210728    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 02 18:00:32.248073 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.248039    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"]
Nov 02 18:00:32.253255 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:00:32.253207    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod92ad0220_435f_4ae8_b743_3421ce98e6fb.slice/crio-2b799dfc861353931928f0368501ec5932f9339bcb26dea69afe9cd496f226f9 WatchSource:0}: Error finding container 2b799dfc861353931928f0368501ec5932f9339bcb26dea69afe9cd496f226f9: Status 404 returned error can't find the container with id 2b799dfc861353931928f0368501ec5932f9339bcb26dea69afe9cd496f226f9
Nov 02 18:00:32.258494 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.257894    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:32.258494 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.257985    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:32.258494 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.258027    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:32.425083 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:00:32.425044    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfae57b7d_d41c_4b47_a4c6_6ff96e936ddb.slice/crio-df821197c29a02246828be44625e753d68d6994a6f8fecb75d865e468b5f10a3 WatchSource:0}: Error finding container df821197c29a02246828be44625e753d68d6994a6f8fecb75d865e468b5f10a3: Status 404 returned error can't find the container with id df821197c29a02246828be44625e753d68d6994a6f8fecb75d865e468b5f10a3
Nov 02 18:00:32.431851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.431505    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"]
Nov 02 18:00:32.955252 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.954892    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs" event={"ID":"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb","Type":"ContainerStarted","Data":"df821197c29a02246828be44625e753d68d6994a6f8fecb75d865e468b5f10a3"}
Nov 02 18:00:32.957442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.957412    4456 generic.go:356] "Generic (PLEG): container finished" podID="32802945-5e87-4339-a7eb-33212aab7ba5" containerID="be2b1174a81ac8ec0f45fb11bddb3dc43efc0e67214ba6610f1874b6ec7eb231" exitCode=0
Nov 02 18:00:32.957743 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.957445    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" event={"ID":"32802945-5e87-4339-a7eb-33212aab7ba5","Type":"ContainerDied","Data":"be2b1174a81ac8ec0f45fb11bddb3dc43efc0e67214ba6610f1874b6ec7eb231"}
Nov 02 18:00:32.958657 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.958635    4456 scope.go:122] "RemoveContainer" containerID="be2b1174a81ac8ec0f45fb11bddb3dc43efc0e67214ba6610f1874b6ec7eb231"
Nov 02 18:00:32.961593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.961557    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:32.961676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.961656    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:32.961714 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.961702    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:32.961938 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.961916    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" event={"ID":"92ad0220-435f-4ae8-b743-3421ce98e6fb","Type":"ContainerStarted","Data":"20b2d3a75919af0722d71b55fa5ed703b6ecda4bdc3481dee9e990f775021807"}
Nov 02 18:00:32.961981 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:32.961965    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" event={"ID":"92ad0220-435f-4ae8-b743-3421ce98e6fb","Type":"ContainerStarted","Data":"2b799dfc861353931928f0368501ec5932f9339bcb26dea69afe9cd496f226f9"}
Nov 02 18:00:33.123393 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.088431    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podStartSLOduration=2.088411645 podStartE2EDuration="2.088411645s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:33.087275365 +0000 UTC m=+1849.578338529" watchObservedRunningTime="2026-11-02 18:00:33.088411645 +0000 UTC m=+1849.579474809"
Nov 02 18:00:33.194104 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.190278    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 02 18:00:33.194104 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.190332    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 02 18:00:33.256836 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.256702    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"]
Nov 02 18:00:33.315143 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.315057    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.327439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.327351    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"]
Nov 02 18:00:33.402907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.395285    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.402907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.395347    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.402907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.395474    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.402907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.395592    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6443b3-b452-4868-9cde-f762cf522284-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.402907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.395618    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6443b3-b452-4868-9cde-f762cf522284-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.402907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.395800    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.402907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.395871    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.402907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.395896    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rhfkr\" (UniqueName: \"kubernetes.io/projected/ef6443b3-b452-4868-9cde-f762cf522284-kube-api-access-rhfkr\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.402907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.395932    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6443b3-b452-4868-9cde-f762cf522284-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.409880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.404683    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:33.551201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.531248    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6443b3-b452-4868-9cde-f762cf522284-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.551201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.531301    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6443b3-b452-4868-9cde-f762cf522284-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.551201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.531473    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.551201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.531523    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.551201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.531550    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rhfkr\" (UniqueName: \"kubernetes.io/projected/ef6443b3-b452-4868-9cde-f762cf522284-kube-api-access-rhfkr\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.551201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.531581    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6443b3-b452-4868-9cde-f762cf522284-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.551201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.531706    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.551201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.531737    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.551201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.531868    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.551201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.532338    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-credential-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.551201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.533238    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6443b3-b452-4868-9cde-f762cf522284-istiod-ca-cert\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.551201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.533452    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-workload-socket\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.552773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.552715    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-istio-envoy\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.555434 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.555408    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-istio-data\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.555870 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.555848    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-workload-certs\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.563875 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.563781    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6443b3-b452-4868-9cde-f762cf522284-istio-token\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.566599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.566513    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6443b3-b452-4868-9cde-f762cf522284-istio-podinfo\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.848961 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.837858    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 18:00:33.848961 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.837936    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 18:00:33.850102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.850018    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rhfkr\" (UniqueName: \"kubernetes.io/projected/ef6443b3-b452-4868-9cde-f762cf522284-kube-api-access-rhfkr\") pod \"e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") " pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:33.862951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:33.862921    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:34.054847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:34.052256    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:34.073342 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:34.073298    4456 generic.go:356] "Generic (PLEG): container finished" podID="fae57b7d-d41c-4b47-a4c6-6ff96e936ddb" containerID="eb0f22e0b2050b9356e1474bc817e81e1dc7dc4d3c6640daa55886540961bf5c" exitCode=0
Nov 02 18:00:34.074082 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:34.073468    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs" event={"ID":"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb","Type":"ContainerDied","Data":"eb0f22e0b2050b9356e1474bc817e81e1dc7dc4d3c6640daa55886540961bf5c"}
Nov 02 18:00:34.076267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:34.076163    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" event={"ID":"32802945-5e87-4339-a7eb-33212aab7ba5","Type":"ContainerStarted","Data":"cae3f46a8bc87e7369f5ef014d19a5821ecaa21b04f97dd5ed71d6ee6eb99acf"}
Nov 02 18:00:34.254856 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:34.237541    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 02 18:00:34.254856 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:34.237608    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 02 18:00:34.265971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:34.265891    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 18:00:34.266265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:34.266238    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 18:00:34.266387 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:34.266301    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 18:00:34.667957 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:34.667923    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"]
Nov 02 18:00:34.793869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:34.787936    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:34.800646 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:34.800598    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:34.800956 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:34.800929    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:00:34.824235 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:34.823167    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 18:00:34.824235 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:34.823217    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 18:00:35.378843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:35.342303    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 18:00:35.378843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:35.342393    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 18:00:35.423726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:35.423687    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:35.460003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:35.459968    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 02 18:00:35.502868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:35.501907    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 02 18:00:35.547879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:35.528970    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" event={"ID":"ef6443b3-b452-4868-9cde-f762cf522284","Type":"ContainerStarted","Data":"6cd905c99e4662f4685dd3c073d95e4f2542965be0683c159b28a2a71070a147"}
Nov 02 18:00:35.892841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:35.890145    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 18:00:35.892841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:35.890231    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 18:00:36.221144 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:36.221042    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 02 18:00:36.233444 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:36.233402    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 02 18:00:36.274977 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:36.274930    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 18:00:36.332856 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:36.330193    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 18:00:36.459897 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:36.459843    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:36.527898 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:36.516615    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:36.750124 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:36.725991    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 18:00:36.815887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:36.803860    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:36.865851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:36.821543    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:36.865851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:36.821964    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 18:00:36.865851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:36.822011    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 18:00:37.003841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:36.995407    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" event={"ID":"ef6443b3-b452-4868-9cde-f762cf522284","Type":"ContainerStarted","Data":"4a335e08a5bc2d765dbc826bbda03d9b544544393a5d585daa44175433b85831"}
Nov 02 18:00:37.026510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.022947    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:37.033502 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.033458    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:37.126522 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.075348    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:37.179866 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.171215    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 02 18:00:37.241857 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.230716    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:37.241857 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.233078    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 18:00:37.261378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.261193    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused" start-of-body=
Nov 02 18:00:37.261378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.261253    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": dial tcp 10.134.0.216:15021: connect: connection refused"
Nov 02 18:00:37.329861 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.312694    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podStartSLOduration=4.312674696 podStartE2EDuration="4.312674696s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:37.073683485 +0000 UTC m=+1853.564746673" watchObservedRunningTime="2026-11-02 18:00:37.312674696 +0000 UTC m=+1853.803737859"
Nov 02 18:00:37.334084 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.326011    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 18:00:37.334451 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.334373    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 18:00:37.334613 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.171277    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 02 18:00:37.424023 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.415448    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:37.452292 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.452258    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 18:00:37.649704 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.649666    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 18:00:37.654020 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.653985    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:37.721909 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.721873    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:37.722627 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.722599    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:37.829909 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.818265    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:37.847842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.826950    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 18:00:37.871516 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.842335    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 18:00:37.871895 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.871848    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 18:00:37.878995 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.878917    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:37.966318 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.960396    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:37.996836 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:37.980877    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 18:00:38.073196 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.037302    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"
Nov 02 18:00:38.073462 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.046949    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 18:00:38.111900 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.111793    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 02 18:00:38.111900 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.111867    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 02 18:00:38.133043 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.133006    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 18:00:38.133881 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.133862    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 18:00:38.166839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.159546    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 18:00:38.285901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.273805    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:38.285901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.274808    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:00:38.382886 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.317091    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 18:00:38.418841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.411146    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 18:00:38.440845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.439066    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-kube-api-access-5vqf9\" (UniqueName: \"kubernetes.io/projected/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-kube-api-access-5vqf9\") pod \"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb\" (UID: \"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb\") "
Nov 02 18:00:38.440845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.439173    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-maas-api-xks-ca\") pod \"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb\" (UID: \"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb\") "
Nov 02 18:00:38.440845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.439372    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-maas-api-openshift-ca\") pod \"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb\" (UID: \"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb\") "
Nov 02 18:00:38.506372 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.480209    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:38.506372 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.497543    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-maas-api-openshift-ca" pod "fae57b7d-d41c-4b47-a4c6-6ff96e936ddb" (UID: "fae57b7d-d41c-4b47-a4c6-6ff96e936ddb"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:38.535545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.535501    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:38.671966 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.671900    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-kube-api-access-5vqf9" pod "fae57b7d-d41c-4b47-a4c6-6ff96e936ddb" (UID: "fae57b7d-d41c-4b47-a4c6-6ff96e936ddb"). InnerVolumeSpecName "kube-api-access-5vqf9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:38.672998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.672971    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-maas-api-xks-ca" pod "fae57b7d-d41c-4b47-a4c6-6ff96e936ddb" (UID: "fae57b7d-d41c-4b47-a4c6-6ff96e936ddb"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:00:38.677013 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.676986    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5vqf9\" (UniqueName: \"kubernetes.io/projected/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-kube-api-access-5vqf9\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:38.677191 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.677177    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:38.678893 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.678863    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:38.690964 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.682913    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:38.690964 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.683193    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 18:00:38.690964 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.684200    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:38.747337 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.747283    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:38.749195 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.749166    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 18:00:38.821331 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.821263    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:38.836104 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.836066    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 18:00:38.844427 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.844388    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs" event={"ID":"fae57b7d-d41c-4b47-a4c6-6ff96e936ddb","Type":"ContainerDied","Data":"df821197c29a02246828be44625e753d68d6994a6f8fecb75d865e468b5f10a3"}
Nov 02 18:00:38.884294 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.884266    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="df821197c29a02246828be44625e753d68d6994a6f8fecb75d865e468b5f10a3"
Nov 02 18:00:38.911854 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.883301    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:00:38.927748 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.869929    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 18:00:38.929305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.893414    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 18:00:38.929305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.928559    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 18:00:38.929305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.916999    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:00:38.929305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.928680    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 18:00:38.929305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.928720    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 18:00:38.949048 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.910399    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"
Nov 02 18:00:38.979438 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:38.979253    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 18:00:39.070385 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:39.068422    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 18:00:39.075900 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:39.075188    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 18:00:39.124069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:39.124010    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 18:00:39.124338 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:39.124315    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 02 18:00:39.124761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:39.124532    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 02 18:00:39.128538 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:39.128500    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 18:00:39.281841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:39.274089    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:39.310171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:39.304102    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 18:00:39.310171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:39.304164    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 18:00:39.432942 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:39.415394    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:39.746294 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:39.694832    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:39.847483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:39.836491    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 18:00:39.847483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:39.836549    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 18:00:40.204510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.200699    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:40.241871 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.241005    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 02 18:00:40.241871 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.241074    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 02 18:00:40.242956 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.242912    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 18:00:40.318909 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.281935    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:40.320192 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.319446    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:40.321138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.321105    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Nov 02 18:00:40.332011 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.329603    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 18:00:40.332011 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.329665    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 18:00:40.548113 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.547559    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:40.551727 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.551673    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 18:00:40.558261 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.558220    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 18:00:40.602845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.580119    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:40.729870 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.714806    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:40.730594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.730558    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:40.843893 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.840503    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 18:00:40.896005 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.895944    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused" start-of-body=
Nov 02 18:00:40.896221 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.896044    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": dial tcp 10.134.0.217:15021: connect: connection refused"
Nov 02 18:00:40.908420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:40.908385    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:41.117899 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:41.097086    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 18:00:41.117899 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:41.097163    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 02 18:00:41.117899 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:41.097208    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 02 18:00:41.136468 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:41.136422    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:41.192967 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:41.192925    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:41.200938 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:41.200900    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:00:41.201786 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:41.201762    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 18:00:41.356511 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:41.356454    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 18:00:41.391641 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:41.391544    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 18:00:41.393275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:41.393237    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 18:00:41.463554 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:41.463357    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 18:00:41.670408 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:41.601339    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused" start-of-body=
Nov 02 18:00:41.670659 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:41.670629    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": dial tcp 10.134.0.188:15021: connect: connection refused"
Nov 02 18:00:41.755192 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:41.755152    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 18:00:42.050399 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:42.049873    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 18:00:42.050399 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:42.049948    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:00:42.050399 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:42.050024    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:42.050399 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:42.050074    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 18:00:42.050399 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:42.050109    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 18:00:42.053341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:42.053312    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:42.054427 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:42.054401    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:00:42.055861 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:42.055785    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 02 18:00:42.057972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:42.056581    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 02 18:00:42.442007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:42.441947    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 18:00:42.475654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:42.472944    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 18:00:42.568314 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:42.568176    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:43.053098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:43.053051    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 02 18:00:43.053609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:43.053114    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 02 18:00:44.053263 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:44.053219    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 02 18:00:44.053674 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:44.053288    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 02 18:00:44.267843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:44.266523    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:45.086020 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:45.065723    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused" start-of-body=
Nov 02 18:00:45.086020 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:45.065802    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": dial tcp 10.134.0.219:15021: connect: connection refused"
Nov 02 18:00:46.127846 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:46.127122    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:46.153851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:46.152396    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:46.181663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:46.180323    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:00:46.330166 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:46.330126    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:46.537158 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:46.537106    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:00:47.415092 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:47.415059    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:47.944798 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:47.944761    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"]
Nov 02 18:00:47.946358 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:47.946334    4456 cpu_manager.go:414] "containerMap: removing container" podUID="fae57b7d-d41c-4b47-a4c6-6ff96e936ddb" containerName="revoke-keys"
Nov 02 18:00:47.946436 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:47.946363    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fae57b7d-d41c-4b47-a4c6-6ff96e936ddb" containerName="revoke-keys"
Nov 02 18:00:47.946594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:47.946580    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="fae57b7d-d41c-4b47-a4c6-6ff96e936ddb" containerName="revoke-keys"
Nov 02 18:00:47.949781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:47.949751    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"
Nov 02 18:00:47.965616 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:47.965586    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:00:47.965781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:47.965715    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:00:47.966784 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:47.966708    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"]
Nov 02 18:00:48.067586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:48.067547    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6773e5d1-439f-46b2-b48f-eb8cc957566d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80911fb3-xrwb7\" (UID: \"6773e5d1-439f-46b2-b48f-eb8cc957566d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"
Nov 02 18:00:48.068280 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:48.067772    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6773e5d1-439f-46b2-b48f-eb8cc957566d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80911fb3-xrwb7\" (UID: \"6773e5d1-439f-46b2-b48f-eb8cc957566d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"
Nov 02 18:00:48.068280 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:48.067963    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-dkx5s\" (UniqueName: \"kubernetes.io/projected/6773e5d1-439f-46b2-b48f-eb8cc957566d-kube-api-access-dkx5s\") pod \"maas-api-revoke-sub-80911fb3-xrwb7\" (UID: \"6773e5d1-439f-46b2-b48f-eb8cc957566d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"
Nov 02 18:00:48.173807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:48.173766    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6773e5d1-439f-46b2-b48f-eb8cc957566d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80911fb3-xrwb7\" (UID: \"6773e5d1-439f-46b2-b48f-eb8cc957566d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"
Nov 02 18:00:48.202471 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:48.202374    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-dkx5s\" (UniqueName: \"kubernetes.io/projected/6773e5d1-439f-46b2-b48f-eb8cc957566d-kube-api-access-dkx5s\") pod \"maas-api-revoke-sub-80911fb3-xrwb7\" (UID: \"6773e5d1-439f-46b2-b48f-eb8cc957566d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"
Nov 02 18:00:48.202699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:48.202676    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6773e5d1-439f-46b2-b48f-eb8cc957566d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80911fb3-xrwb7\" (UID: \"6773e5d1-439f-46b2-b48f-eb8cc957566d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"
Nov 02 18:00:48.213430 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:48.213393    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6773e5d1-439f-46b2-b48f-eb8cc957566d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-80911fb3-xrwb7\" (UID: \"6773e5d1-439f-46b2-b48f-eb8cc957566d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"
Nov 02 18:00:48.218476 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:48.218442    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:48.233520 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:48.233486    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6773e5d1-439f-46b2-b48f-eb8cc957566d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-80911fb3-xrwb7\" (UID: \"6773e5d1-439f-46b2-b48f-eb8cc957566d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"
Nov 02 18:00:48.236300 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:48.236266    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-dkx5s\" (UniqueName: \"kubernetes.io/projected/6773e5d1-439f-46b2-b48f-eb8cc957566d-kube-api-access-dkx5s\") pod \"maas-api-revoke-sub-80911fb3-xrwb7\" (UID: \"6773e5d1-439f-46b2-b48f-eb8cc957566d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"
Nov 02 18:00:48.268297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:48.268255    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"
Nov 02 18:00:48.486096 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:48.486006    4456 scope.go:122] "RemoveContainer" containerID="aedc3f687f79a037c431dd3d6319190e8a3019b2726ccc734c9a41664b91d46f"
Nov 02 18:00:48.521792 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:48.521768    4456 scope.go:122] "RemoveContainer" containerID="d376b250cd604ac4f7fa5e98f47fea37e0b793fc6d46a3b9eb9b4ab25fce9a79"
Nov 02 18:00:48.818598 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:48.818574    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"]
Nov 02 18:00:48.826660 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:00:48.826539    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6773e5d1_439f_46b2_b48f_eb8cc957566d.slice/crio-68e5a283fa00c895c0b0527d44d5dd9d8b15fe685096b94ce28c201246809ab0 WatchSource:0}: Error finding container 68e5a283fa00c895c0b0527d44d5dd9d8b15fe685096b94ce28c201246809ab0: Status 404 returned error can't find the container with id 68e5a283fa00c895c0b0527d44d5dd9d8b15fe685096b94ce28c201246809ab0
Nov 02 18:00:49.296151 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:49.296047    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7" event={"ID":"6773e5d1-439f-46b2-b48f-eb8cc957566d","Type":"ContainerStarted","Data":"0afcedd1deec4a2d5cae278ee100bd19f987c5e216a88b6ff7615e95ad5c8ed5"}
Nov 02 18:00:49.296345 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:49.296210    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7" event={"ID":"6773e5d1-439f-46b2-b48f-eb8cc957566d","Type":"ContainerStarted","Data":"68e5a283fa00c895c0b0527d44d5dd9d8b15fe685096b94ce28c201246809ab0"}
Nov 02 18:00:49.346320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:49.346263    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7" podStartSLOduration=2.346248955 podStartE2EDuration="2.346248955s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:00:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:00:49.337590352 +0000 UTC m=+1865.828653547" watchObservedRunningTime="2026-11-02 18:00:49.346248955 +0000 UTC m=+1865.837312116"
Nov 02 18:00:49.849760 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:49.849726    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:50.310211 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:50.310174    4456 generic.go:356] "Generic (PLEG): container finished" podID="6773e5d1-439f-46b2-b48f-eb8cc957566d" containerID="0afcedd1deec4a2d5cae278ee100bd19f987c5e216a88b6ff7615e95ad5c8ed5" exitCode=0
Nov 02 18:00:50.310408 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:50.310259    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7" event={"ID":"6773e5d1-439f-46b2-b48f-eb8cc957566d","Type":"ContainerDied","Data":"0afcedd1deec4a2d5cae278ee100bd19f987c5e216a88b6ff7615e95ad5c8ed5"}
Nov 02 18:00:50.624787 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:50.624701    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"]
Nov 02 18:00:50.625045 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:50.624963    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v" podUID="f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" containerName="istio-proxy" containerID="cri-o://59f84b293d5554412b761fd7de2e57d2b60039eae14a8b53dbeaa239477d67bd" gracePeriod=30
Nov 02 18:00:51.518439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:51.518415    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"
Nov 02 18:00:51.587318 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:51.587087    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6773e5d1-439f-46b2-b48f-eb8cc957566d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6773e5d1-439f-46b2-b48f-eb8cc957566d-maas-api-xks-ca\") pod \"6773e5d1-439f-46b2-b48f-eb8cc957566d\" (UID: \"6773e5d1-439f-46b2-b48f-eb8cc957566d\") "
Nov 02 18:00:51.587318 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:51.587183    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6773e5d1-439f-46b2-b48f-eb8cc957566d-kube-api-access-dkx5s\" (UniqueName: \"kubernetes.io/projected/6773e5d1-439f-46b2-b48f-eb8cc957566d-kube-api-access-dkx5s\") pod \"6773e5d1-439f-46b2-b48f-eb8cc957566d\" (UID: \"6773e5d1-439f-46b2-b48f-eb8cc957566d\") "
Nov 02 18:00:51.587567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:51.587503    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6773e5d1-439f-46b2-b48f-eb8cc957566d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6773e5d1-439f-46b2-b48f-eb8cc957566d-maas-api-openshift-ca\") pod \"6773e5d1-439f-46b2-b48f-eb8cc957566d\" (UID: \"6773e5d1-439f-46b2-b48f-eb8cc957566d\") "
Nov 02 18:00:51.588887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:51.588853    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6773e5d1-439f-46b2-b48f-eb8cc957566d-maas-api-openshift-ca" pod "6773e5d1-439f-46b2-b48f-eb8cc957566d" (UID: "6773e5d1-439f-46b2-b48f-eb8cc957566d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:51.592688 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:51.592665    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6773e5d1-439f-46b2-b48f-eb8cc957566d-kube-api-access-dkx5s" pod "6773e5d1-439f-46b2-b48f-eb8cc957566d" (UID: "6773e5d1-439f-46b2-b48f-eb8cc957566d"). InnerVolumeSpecName "kube-api-access-dkx5s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:51.593635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:51.592731    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6773e5d1-439f-46b2-b48f-eb8cc957566d-maas-api-xks-ca" pod "6773e5d1-439f-46b2-b48f-eb8cc957566d" (UID: "6773e5d1-439f-46b2-b48f-eb8cc957566d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:00:51.634632 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:51.634602    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:51.692838 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:51.692154    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dkx5s\" (UniqueName: \"kubernetes.io/projected/6773e5d1-439f-46b2-b48f-eb8cc957566d-kube-api-access-dkx5s\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:51.692838 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:51.692195    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6773e5d1-439f-46b2-b48f-eb8cc957566d-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:51.692838 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:51.692211    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6773e5d1-439f-46b2-b48f-eb8cc957566d-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:52.330724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:52.330688    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7" event={"ID":"6773e5d1-439f-46b2-b48f-eb8cc957566d","Type":"ContainerDied","Data":"68e5a283fa00c895c0b0527d44d5dd9d8b15fe685096b94ce28c201246809ab0"}
Nov 02 18:00:52.332167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:52.330764    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="68e5a283fa00c895c0b0527d44d5dd9d8b15fe685096b94ce28c201246809ab0"
Nov 02 18:00:52.332167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:52.330706    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"
Nov 02 18:00:53.359743 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:53.359687    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:55.395839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:55.395667    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:55.972581 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:55.972553    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 18:00:56.122465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.122424    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-podinfo\") pod \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") "
Nov 02 18:00:56.123465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.122567    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-kube-api-access-fgjrd\" (UniqueName: \"kubernetes.io/projected/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-kube-api-access-fgjrd\") pod \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") "
Nov 02 18:00:56.123465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.122791    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-credential-socket\") pod \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") "
Nov 02 18:00:56.123465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.122925    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-data\") pod \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") "
Nov 02 18:00:56.123465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.122958    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istiod-ca-cert\") pod \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") "
Nov 02 18:00:56.123465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.123111    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-workload-socket\") pod \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") "
Nov 02 18:00:56.123465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.123142    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-envoy\") pod \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") "
Nov 02 18:00:56.123465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.123215    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-token\" (UniqueName: \"kubernetes.io/projected/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-token\") pod \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") "
Nov 02 18:00:56.123465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.123260    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-workload-certs\") pod \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\" (UID: \"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3\") "
Nov 02 18:00:56.148676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.148638    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-credential-socket" pod "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" (UID: "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:56.148891 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.148868    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-workload-socket" pod "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" (UID: "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:56.148977 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.148900    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-workload-certs" pod "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" (UID: "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:56.157519 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.157490    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-podinfo" pod "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" (UID: "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:00:56.157656 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.157516    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-data" pod "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" (UID: "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:56.157860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.157803    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istiod-ca-cert" pod "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" (UID: "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:00:56.157983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.157865    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-kube-api-access-fgjrd" pod "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" (UID: "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3"). InnerVolumeSpecName "kube-api-access-fgjrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:56.160174 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.160154    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-token" pod "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" (UID: "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:00:56.163478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.163450    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-envoy" pod "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" (UID: "f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:00:56.225847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.225799    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:56.225847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.225840    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:56.225847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.225850    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:56.225847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.225858    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:56.226694 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.225867    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:56.226694 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.225875    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:56.226694 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.225882    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:56.226694 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.225890    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fgjrd\" (UniqueName: \"kubernetes.io/projected/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-kube-api-access-fgjrd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:56.226694 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.225898    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:00:56.369958 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.369925    4456 generic.go:356] "Generic (PLEG): container finished" podID="f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" containerID="59f84b293d5554412b761fd7de2e57d2b60039eae14a8b53dbeaa239477d67bd" exitCode=0
Nov 02 18:00:56.370133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.369967    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"
Nov 02 18:00:56.370133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.370007    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v" event={"ID":"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3","Type":"ContainerDied","Data":"59f84b293d5554412b761fd7de2e57d2b60039eae14a8b53dbeaa239477d67bd"}
Nov 02 18:00:56.370133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.370067    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v" event={"ID":"f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3","Type":"ContainerDied","Data":"cdd80a3f67466a8e118c64c734d288d17970ba233eefdf4a96fcb78951cac647"}
Nov 02 18:00:56.370133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.370095    4456 scope.go:122] "RemoveContainer" containerID="59f84b293d5554412b761fd7de2e57d2b60039eae14a8b53dbeaa239477d67bd"
Nov 02 18:00:56.391932 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.391908    4456 scope.go:122] "RemoveContainer" containerID="59f84b293d5554412b761fd7de2e57d2b60039eae14a8b53dbeaa239477d67bd"
Nov 02 18:00:56.392285 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:00:56.392259    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"59f84b293d5554412b761fd7de2e57d2b60039eae14a8b53dbeaa239477d67bd\": container with ID starting with 59f84b293d5554412b761fd7de2e57d2b60039eae14a8b53dbeaa239477d67bd not found: ID does not exist" containerID="59f84b293d5554412b761fd7de2e57d2b60039eae14a8b53dbeaa239477d67bd"
Nov 02 18:00:56.392375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.392299    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"59f84b293d5554412b761fd7de2e57d2b60039eae14a8b53dbeaa239477d67bd"} err="failed to get container status \"59f84b293d5554412b761fd7de2e57d2b60039eae14a8b53dbeaa239477d67bd\": rpc error: code = NotFound desc = could not find container \"59f84b293d5554412b761fd7de2e57d2b60039eae14a8b53dbeaa239477d67bd\": container with ID starting with 59f84b293d5554412b761fd7de2e57d2b60039eae14a8b53dbeaa239477d67bd not found: ID does not exist"
Nov 02 18:00:56.428645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.428613    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"]
Nov 02 18:00:56.439310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:56.439283    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-bz24v"]
Nov 02 18:00:57.011876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:57.011797    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:00:57.620185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:57.620148    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" path="/var/lib/kubelet/pods/f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3/volumes"
Nov 02 18:00:58.764307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:00:58.764272    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:00.319697 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:00.319663    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:02.012552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:02.012512    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:03.674084 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:03.673991    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:05.211311 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:05.211268    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:06.493522 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:06.493465    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:06.873535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:06.873460    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"]
Nov 02 18:01:06.874418 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:06.874400    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" containerName="istio-proxy"
Nov 02 18:01:06.875219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:06.874419    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" containerName="istio-proxy"
Nov 02 18:01:06.875219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:06.874436    4456 cpu_manager.go:414] "containerMap: removing container" podUID="6773e5d1-439f-46b2-b48f-eb8cc957566d" containerName="revoke-keys"
Nov 02 18:01:06.875219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:06.874441    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6773e5d1-439f-46b2-b48f-eb8cc957566d" containerName="revoke-keys"
Nov 02 18:01:06.875219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:06.874585    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f072ed0e-d8c7-48ea-973b-9b09b1b1f0d3" containerName="istio-proxy"
Nov 02 18:01:06.875219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:06.874594    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6773e5d1-439f-46b2-b48f-eb8cc957566d" containerName="revoke-keys"
Nov 02 18:01:06.877784 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:06.877769    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"
Nov 02 18:01:06.897550 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:06.897522    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"]
Nov 02 18:01:06.903404 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:06.903377    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:01:06.903622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:06.903607    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:01:07.046305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:07.046267    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-s8t95\" (UniqueName: \"kubernetes.io/projected/b635baf6-b8b5-464f-8ac1-b8d66b38931a-kube-api-access-s8t95\") pod \"maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj\" (UID: \"b635baf6-b8b5-464f-8ac1-b8d66b38931a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"
Nov 02 18:01:07.046476 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:07.046395    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b635baf6-b8b5-464f-8ac1-b8d66b38931a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj\" (UID: \"b635baf6-b8b5-464f-8ac1-b8d66b38931a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"
Nov 02 18:01:07.046595 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:07.046578    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b635baf6-b8b5-464f-8ac1-b8d66b38931a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj\" (UID: \"b635baf6-b8b5-464f-8ac1-b8d66b38931a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"
Nov 02 18:01:07.080609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:07.080553    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:07.150841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:07.150709    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b635baf6-b8b5-464f-8ac1-b8d66b38931a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj\" (UID: \"b635baf6-b8b5-464f-8ac1-b8d66b38931a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"
Nov 02 18:01:07.157178 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:07.151420    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-s8t95\" (UniqueName: \"kubernetes.io/projected/b635baf6-b8b5-464f-8ac1-b8d66b38931a-kube-api-access-s8t95\") pod \"maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj\" (UID: \"b635baf6-b8b5-464f-8ac1-b8d66b38931a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"
Nov 02 18:01:07.157512 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:07.157485    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b635baf6-b8b5-464f-8ac1-b8d66b38931a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj\" (UID: \"b635baf6-b8b5-464f-8ac1-b8d66b38931a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"
Nov 02 18:01:07.159398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:07.159317    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b635baf6-b8b5-464f-8ac1-b8d66b38931a-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj\" (UID: \"b635baf6-b8b5-464f-8ac1-b8d66b38931a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"
Nov 02 18:01:07.172842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:07.169838    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b635baf6-b8b5-464f-8ac1-b8d66b38931a-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj\" (UID: \"b635baf6-b8b5-464f-8ac1-b8d66b38931a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"
Nov 02 18:01:07.186629 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:07.186597    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-s8t95\" (UniqueName: \"kubernetes.io/projected/b635baf6-b8b5-464f-8ac1-b8d66b38931a-kube-api-access-s8t95\") pod \"maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj\" (UID: \"b635baf6-b8b5-464f-8ac1-b8d66b38931a\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"
Nov 02 18:01:07.193283 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:07.193258    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"
Nov 02 18:01:07.525343 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:07.525311    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"]
Nov 02 18:01:07.550154 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:01:07.550121    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podb635baf6_b8b5_464f_8ac1_b8d66b38931a.slice/crio-383683489bdd7b9e96791e98b32113b045c87e8c24bf3712462486398736f759 WatchSource:0}: Error finding container 383683489bdd7b9e96791e98b32113b045c87e8c24bf3712462486398736f759: Status 404 returned error can't find the container with id 383683489bdd7b9e96791e98b32113b045c87e8c24bf3712462486398736f759
Nov 02 18:01:08.516098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:08.516055    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj" event={"ID":"b635baf6-b8b5-464f-8ac1-b8d66b38931a","Type":"ContainerStarted","Data":"6e1456e56a48150e86433f5c057ed17ac8273504ff8c6704b7f0bb813f92acd6"}
Nov 02 18:01:08.516306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:08.516137    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj" event={"ID":"b635baf6-b8b5-464f-8ac1-b8d66b38931a","Type":"ContainerStarted","Data":"383683489bdd7b9e96791e98b32113b045c87e8c24bf3712462486398736f759"}
Nov 02 18:01:08.558931 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:08.558886    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj" podStartSLOduration=2.558872993 podStartE2EDuration="2.558872993s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:08.557079685 +0000 UTC m=+1885.048142868" watchObservedRunningTime="2026-11-02 18:01:08.558872993 +0000 UTC m=+1885.049936154"
Nov 02 18:01:08.841751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:08.841677    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:09.530920 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:09.530881    4456 generic.go:356] "Generic (PLEG): container finished" podID="b635baf6-b8b5-464f-8ac1-b8d66b38931a" containerID="6e1456e56a48150e86433f5c057ed17ac8273504ff8c6704b7f0bb813f92acd6" exitCode=0
Nov 02 18:01:09.531132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:09.530975    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj" event={"ID":"b635baf6-b8b5-464f-8ac1-b8d66b38931a","Type":"ContainerDied","Data":"6e1456e56a48150e86433f5c057ed17ac8273504ff8c6704b7f0bb813f92acd6"}
Nov 02 18:01:10.504921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:10.504871    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:10.754092 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:10.754068    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"
Nov 02 18:01:10.854590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:10.854507    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/b635baf6-b8b5-464f-8ac1-b8d66b38931a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b635baf6-b8b5-464f-8ac1-b8d66b38931a-maas-api-xks-ca\") pod \"b635baf6-b8b5-464f-8ac1-b8d66b38931a\" (UID: \"b635baf6-b8b5-464f-8ac1-b8d66b38931a\") "
Nov 02 18:01:10.854802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:10.854784    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b635baf6-b8b5-464f-8ac1-b8d66b38931a-kube-api-access-s8t95\" (UniqueName: \"kubernetes.io/projected/b635baf6-b8b5-464f-8ac1-b8d66b38931a-kube-api-access-s8t95\") pod \"b635baf6-b8b5-464f-8ac1-b8d66b38931a\" (UID: \"b635baf6-b8b5-464f-8ac1-b8d66b38931a\") "
Nov 02 18:01:10.854934 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:10.854901    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b635baf6-b8b5-464f-8ac1-b8d66b38931a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b635baf6-b8b5-464f-8ac1-b8d66b38931a-maas-api-openshift-ca\") pod \"b635baf6-b8b5-464f-8ac1-b8d66b38931a\" (UID: \"b635baf6-b8b5-464f-8ac1-b8d66b38931a\") "
Nov 02 18:01:10.855303 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:10.855223    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b635baf6-b8b5-464f-8ac1-b8d66b38931a-maas-api-openshift-ca" pod "b635baf6-b8b5-464f-8ac1-b8d66b38931a" (UID: "b635baf6-b8b5-464f-8ac1-b8d66b38931a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:01:10.856318 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:10.856048    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/b635baf6-b8b5-464f-8ac1-b8d66b38931a-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:10.857356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:10.857328    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/b635baf6-b8b5-464f-8ac1-b8d66b38931a-maas-api-xks-ca" pod "b635baf6-b8b5-464f-8ac1-b8d66b38931a" (UID: "b635baf6-b8b5-464f-8ac1-b8d66b38931a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:01:10.857471 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:10.857370    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b635baf6-b8b5-464f-8ac1-b8d66b38931a-kube-api-access-s8t95" pod "b635baf6-b8b5-464f-8ac1-b8d66b38931a" (UID: "b635baf6-b8b5-464f-8ac1-b8d66b38931a"). InnerVolumeSpecName "kube-api-access-s8t95". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:10.958396 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:10.958361    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s8t95\" (UniqueName: \"kubernetes.io/projected/b635baf6-b8b5-464f-8ac1-b8d66b38931a-kube-api-access-s8t95\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:10.958396 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:10.958392    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/b635baf6-b8b5-464f-8ac1-b8d66b38931a-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:11.561118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:11.561086    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"
Nov 02 18:01:11.561516 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:11.561083    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj" event={"ID":"b635baf6-b8b5-464f-8ac1-b8d66b38931a","Type":"ContainerDied","Data":"383683489bdd7b9e96791e98b32113b045c87e8c24bf3712462486398736f759"}
Nov 02 18:01:11.561516 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:11.561226    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="383683489bdd7b9e96791e98b32113b045c87e8c24bf3712462486398736f759"
Nov 02 18:01:12.482464 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:12.482425    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:13.663614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:13.663571    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:14.064695 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:14.062772    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:15.881885 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:15.881802    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:16.496842 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:01:16.487691    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:01:17.579278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:17.579243    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"]
Nov 02 18:01:17.747669 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:17.747063    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:17.943471 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:17.943366    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:19.649458 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:19.649421    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:21.131844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:21.130886    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"]
Nov 02 18:01:21.168925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:21.162142    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4" podUID="712733fd-b069-40c2-b82c-7a86005841c0" containerName="maas-api" containerID="cri-o://928e46cb358a206070aead24b8b5fce080e7a3ef9e1fb8e3d40a9f31b8e52374" gracePeriod=30
Nov 02 18:01:21.248373 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:21.241979    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:21.966460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:21.966383    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:22.092910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.092804    4456 generic.go:356] "Generic (PLEG): container finished" podID="712733fd-b069-40c2-b82c-7a86005841c0" containerID="928e46cb358a206070aead24b8b5fce080e7a3ef9e1fb8e3d40a9f31b8e52374" exitCode=0
Nov 02 18:01:22.093464 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.093436    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4" event={"ID":"712733fd-b069-40c2-b82c-7a86005841c0","Type":"ContainerDied","Data":"928e46cb358a206070aead24b8b5fce080e7a3ef9e1fb8e3d40a9f31b8e52374"}
Nov 02 18:01:22.148332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.148146    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:22.185905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.185859    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"]
Nov 02 18:01:22.215924 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.197866    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-063f48-8x9mj"]
Nov 02 18:01:22.300840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.300617    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:22.631842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.628131    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n" podUID="38cc3415-dd2f-4e7a-91a9-f3740f2cc826" containerName="payload-processing" containerID="cri-o://3ca3b8e733f06e462266217f185ef0da9df2ed08e368a7c11910091f122469d5" gracePeriod=25
Nov 02 18:01:22.709140 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.703136    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:22.759003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.758623    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 18:01:22.777193 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.777164    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/712733fd-b069-40c2-b82c-7a86005841c0-metrics-certs\" (UniqueName: \"kubernetes.io/secret/712733fd-b069-40c2-b82c-7a86005841c0-metrics-certs\") pod \"712733fd-b069-40c2-b82c-7a86005841c0\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") "
Nov 02 18:01:22.777343 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.777213    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/712733fd-b069-40c2-b82c-7a86005841c0-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/712733fd-b069-40c2-b82c-7a86005841c0-openshift-service-ca\") pod \"712733fd-b069-40c2-b82c-7a86005841c0\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") "
Nov 02 18:01:22.777976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.777365    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/712733fd-b069-40c2-b82c-7a86005841c0-kube-api-access-2q76q\" (UniqueName: \"kubernetes.io/projected/712733fd-b069-40c2-b82c-7a86005841c0-kube-api-access-2q76q\") pod \"712733fd-b069-40c2-b82c-7a86005841c0\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") "
Nov 02 18:01:22.777976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.777428    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/712733fd-b069-40c2-b82c-7a86005841c0-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/712733fd-b069-40c2-b82c-7a86005841c0-maas-api-tls\") pod \"712733fd-b069-40c2-b82c-7a86005841c0\" (UID: \"712733fd-b069-40c2-b82c-7a86005841c0\") "
Nov 02 18:01:22.778946 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.778050    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/712733fd-b069-40c2-b82c-7a86005841c0-openshift-service-ca" pod "712733fd-b069-40c2-b82c-7a86005841c0" (UID: "712733fd-b069-40c2-b82c-7a86005841c0"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:01:22.778946 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.778657    4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/712733fd-b069-40c2-b82c-7a86005841c0-openshift-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:22.800036 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.799966    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712733fd-b069-40c2-b82c-7a86005841c0-metrics-certs" pod "712733fd-b069-40c2-b82c-7a86005841c0" (UID: "712733fd-b069-40c2-b82c-7a86005841c0"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:01:22.810552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.808285    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/712733fd-b069-40c2-b82c-7a86005841c0-maas-api-tls" pod "712733fd-b069-40c2-b82c-7a86005841c0" (UID: "712733fd-b069-40c2-b82c-7a86005841c0"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:01:22.833972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.830557    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/712733fd-b069-40c2-b82c-7a86005841c0-kube-api-access-2q76q" pod "712733fd-b069-40c2-b82c-7a86005841c0" (UID: "712733fd-b069-40c2-b82c-7a86005841c0"). InnerVolumeSpecName "kube-api-access-2q76q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:22.954643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.933373    4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/712733fd-b069-40c2-b82c-7a86005841c0-metrics-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:22.954643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.933409    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q76q\" (UniqueName: \"kubernetes.io/projected/712733fd-b069-40c2-b82c-7a86005841c0-kube-api-access-2q76q\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:22.954643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:22.933435    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/712733fd-b069-40c2-b82c-7a86005841c0-maas-api-tls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:23.151336 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:23.151254    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4" event={"ID":"712733fd-b069-40c2-b82c-7a86005841c0","Type":"ContainerDied","Data":"84c56a4b01e8a518c60a2977b79b8f1da0dc24a7e631218a0bf9c99fc4d4e60e"}
Nov 02 18:01:23.151751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:23.151347    4456 scope.go:122] "RemoveContainer" containerID="928e46cb358a206070aead24b8b5fce080e7a3ef9e1fb8e3d40a9f31b8e52374"
Nov 02 18:01:23.151751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:23.151475    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"
Nov 02 18:01:23.251017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:23.250981    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"]
Nov 02 18:01:23.277858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:23.277365    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-inf-a-063f48-76b6cdd555-fpws4"]
Nov 02 18:01:23.630407 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:23.630373    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="712733fd-b069-40c2-b82c-7a86005841c0" path="/var/lib/kubelet/pods/712733fd-b069-40c2-b82c-7a86005841c0/volumes"
Nov 02 18:01:23.631124 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:23.631099    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b635baf6-b8b5-464f-8ac1-b8d66b38931a" path="/var/lib/kubelet/pods/b635baf6-b8b5-464f-8ac1-b8d66b38931a/volumes"
Nov 02 18:01:24.160206 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:24.160163    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:25.856791 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:25.856732    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:27.299341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:27.299300    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n" podUID="38cc3415-dd2f-4e7a-91a9-f3740f2cc826" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:01:27.576436 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:27.576350    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:29.370582 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:29.370255    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:30.173921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:30.173883    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:30.331624 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:30.331533    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:30.660188 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:30.660149    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:30.911978 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:30.911888    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"]
Nov 02 18:01:30.973447 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:30.912459    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l" podUID="ab077054-0cce-44bd-aecc-ee2938d106c9" containerName="istio-proxy" containerID="cri-o://993c47b2e4ab9a8bec71be49a1d715eb16bb30e2cfae42e995f4ea0dc51b11d2" gracePeriod=30
Nov 02 18:01:31.503228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:31.503193    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:31.558686 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:31.558655    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"]
Nov 02 18:01:31.766596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:31.766514    4456 cpu_manager.go:414] "containerMap: removing container" podUID="712733fd-b069-40c2-b82c-7a86005841c0" containerName="maas-api"
Nov 02 18:01:31.766596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:31.766556    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="712733fd-b069-40c2-b82c-7a86005841c0" containerName="maas-api"
Nov 02 18:01:31.828155 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:31.766625    4456 cpu_manager.go:414] "containerMap: removing container" podUID="b635baf6-b8b5-464f-8ac1-b8d66b38931a" containerName="revoke-keys"
Nov 02 18:01:31.828155 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:31.766633    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b635baf6-b8b5-464f-8ac1-b8d66b38931a" containerName="revoke-keys"
Nov 02 18:01:31.828155 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:31.766923    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b635baf6-b8b5-464f-8ac1-b8d66b38931a" containerName="revoke-keys"
Nov 02 18:01:31.828155 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:31.766952    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="712733fd-b069-40c2-b82c-7a86005841c0" containerName="maas-api"
Nov 02 18:01:32.005576 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.005538    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"
Nov 02 18:01:32.054098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.023382    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:01:32.054098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.034038    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:01:32.054098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.034743    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wn9m7\" (UniqueName: \"kubernetes.io/projected/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-kube-api-access-wn9m7\") pod \"maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx\" (UID: \"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"
Nov 02 18:01:32.054098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.034942    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx\" (UID: \"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"
Nov 02 18:01:32.054098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.035053    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx\" (UID: \"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"
Nov 02 18:01:32.323050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.097966    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:01:32.323050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.098012    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:32.323050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.136216    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx\" (UID: \"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"
Nov 02 18:01:32.323050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.136275    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx\" (UID: \"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"
Nov 02 18:01:32.323050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.136477    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wn9m7\" (UniqueName: \"kubernetes.io/projected/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-kube-api-access-wn9m7\") pod \"maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx\" (UID: \"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"
Nov 02 18:01:32.323050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.137087    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx\" (UID: \"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"
Nov 02 18:01:32.323050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.286441    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx\" (UID: \"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"
Nov 02 18:01:32.323050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.286538    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wn9m7\" (UniqueName: \"kubernetes.io/projected/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-kube-api-access-wn9m7\") pod \"maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx\" (UID: \"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"
Nov 02 18:01:32.373417 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.344723    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"]
Nov 02 18:01:32.425732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:32.425699    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"
Nov 02 18:01:33.638707 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:33.638663    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:34.379547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:34.207980    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:34.401718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:34.401688    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"]
Nov 02 18:01:34.483349 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:34.483317    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx" event={"ID":"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0","Type":"ContainerStarted","Data":"697d8b131e833198d552548be152fe68ff07b10ff277142ebc91e224422bdfec"}
Nov 02 18:01:35.520199 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:35.520158    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx" event={"ID":"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0","Type":"ContainerStarted","Data":"d9c0ba7de1c37929a22a23628e19d8a7160ddd083595efe4618b8976b53a7039"}
Nov 02 18:01:35.543051 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:35.542980    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:35.567996 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:35.567941    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx" podStartSLOduration=4.567921998 podStartE2EDuration="4.567921998s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:31 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:35.564138393 +0000 UTC m=+1912.055201555" watchObservedRunningTime="2026-11-02 18:01:35.567921998 +0000 UTC m=+1912.058985160"
Nov 02 18:01:36.271601 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:36.271226    4456 patch_prober.go:29] interesting pod/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:01:36.271601 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:36.271309    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l" podUID="ab077054-0cce-44bd-aecc-ee2938d106c9" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:01:36.535465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:36.535378    4456 generic.go:356] "Generic (PLEG): container finished" podID="e367b7a1-9c38-451a-a8dc-e8b8d2698fb0" containerID="d9c0ba7de1c37929a22a23628e19d8a7160ddd083595efe4618b8976b53a7039" exitCode=0
Nov 02 18:01:36.535880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:36.535472    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx" event={"ID":"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0","Type":"ContainerDied","Data":"d9c0ba7de1c37929a22a23628e19d8a7160ddd083595efe4618b8976b53a7039"}
Nov 02 18:01:36.661729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:36.661665    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:37.245166 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.245081    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n" podUID="38cc3415-dd2f-4e7a-91a9-f3740f2cc826" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:01:37.382003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.381969    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 18:01:37.437372 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.437331    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-envoy\") pod \"ab077054-0cce-44bd-aecc-ee2938d106c9\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") "
Nov 02 18:01:37.439363 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.437515    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-token\" (UniqueName: \"kubernetes.io/projected/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-token\") pod \"ab077054-0cce-44bd-aecc-ee2938d106c9\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") "
Nov 02 18:01:37.439363 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.437557    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-workload-socket\") pod \"ab077054-0cce-44bd-aecc-ee2938d106c9\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") "
Nov 02 18:01:37.439363 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.437614    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-podinfo\") pod \"ab077054-0cce-44bd-aecc-ee2938d106c9\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") "
Nov 02 18:01:37.439363 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.437652    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-credential-socket\") pod \"ab077054-0cce-44bd-aecc-ee2938d106c9\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") "
Nov 02 18:01:37.439363 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.437780    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ab077054-0cce-44bd-aecc-ee2938d106c9-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab077054-0cce-44bd-aecc-ee2938d106c9-istiod-ca-cert\") pod \"ab077054-0cce-44bd-aecc-ee2938d106c9\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") "
Nov 02 18:01:37.439363 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.437887    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-data\") pod \"ab077054-0cce-44bd-aecc-ee2938d106c9\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") "
Nov 02 18:01:37.439363 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.438035    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-workload-certs\") pod \"ab077054-0cce-44bd-aecc-ee2938d106c9\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") "
Nov 02 18:01:37.439363 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.438144    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ab077054-0cce-44bd-aecc-ee2938d106c9-kube-api-access-2q74n\" (UniqueName: \"kubernetes.io/projected/ab077054-0cce-44bd-aecc-ee2938d106c9-kube-api-access-2q74n\") pod \"ab077054-0cce-44bd-aecc-ee2938d106c9\" (UID: \"ab077054-0cce-44bd-aecc-ee2938d106c9\") "
Nov 02 18:01:37.447851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.443347    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-workload-socket" pod "ab077054-0cce-44bd-aecc-ee2938d106c9" (UID: "ab077054-0cce-44bd-aecc-ee2938d106c9"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:37.447851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.444501    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-credential-socket" pod "ab077054-0cce-44bd-aecc-ee2938d106c9" (UID: "ab077054-0cce-44bd-aecc-ee2938d106c9"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:37.447851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.445004    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-data" pod "ab077054-0cce-44bd-aecc-ee2938d106c9" (UID: "ab077054-0cce-44bd-aecc-ee2938d106c9"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:37.448650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.448606    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ab077054-0cce-44bd-aecc-ee2938d106c9-istiod-ca-cert" pod "ab077054-0cce-44bd-aecc-ee2938d106c9" (UID: "ab077054-0cce-44bd-aecc-ee2938d106c9"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:01:37.451379 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.450958    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-token" pod "ab077054-0cce-44bd-aecc-ee2938d106c9" (UID: "ab077054-0cce-44bd-aecc-ee2938d106c9"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:37.451379 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.450995    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ab077054-0cce-44bd-aecc-ee2938d106c9-kube-api-access-2q74n" pod "ab077054-0cce-44bd-aecc-ee2938d106c9" (UID: "ab077054-0cce-44bd-aecc-ee2938d106c9"). InnerVolumeSpecName "kube-api-access-2q74n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:37.454004 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.453975    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-podinfo" pod "ab077054-0cce-44bd-aecc-ee2938d106c9" (UID: "ab077054-0cce-44bd-aecc-ee2938d106c9"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:01:37.454108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.453982    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-envoy" pod "ab077054-0cce-44bd-aecc-ee2938d106c9" (UID: "ab077054-0cce-44bd-aecc-ee2938d106c9"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:37.454108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.454015    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-workload-certs" pod "ab077054-0cce-44bd-aecc-ee2938d106c9" (UID: "ab077054-0cce-44bd-aecc-ee2938d106c9"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:37.525891 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.522593    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:37.612718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.612683    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:37.612718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.612716    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2q74n\" (UniqueName: \"kubernetes.io/projected/ab077054-0cce-44bd-aecc-ee2938d106c9-kube-api-access-2q74n\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:37.613351 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.612730    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:37.613351 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.612742    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:37.613351 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.612753    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:37.613351 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.612763    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:37.613351 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.612777    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:37.613351 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.612790    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ab077054-0cce-44bd-aecc-ee2938d106c9-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:37.613351 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.612802    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ab077054-0cce-44bd-aecc-ee2938d106c9-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:37.622410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.622369    4456 generic.go:356] "Generic (PLEG): container finished" podID="ab077054-0cce-44bd-aecc-ee2938d106c9" containerID="993c47b2e4ab9a8bec71be49a1d715eb16bb30e2cfae42e995f4ea0dc51b11d2" exitCode=0
Nov 02 18:01:37.623108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.623081    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"
Nov 02 18:01:37.636558 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.634326    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l" event={"ID":"ab077054-0cce-44bd-aecc-ee2938d106c9","Type":"ContainerDied","Data":"993c47b2e4ab9a8bec71be49a1d715eb16bb30e2cfae42e995f4ea0dc51b11d2"}
Nov 02 18:01:37.636686 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.636610    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l" event={"ID":"ab077054-0cce-44bd-aecc-ee2938d106c9","Type":"ContainerDied","Data":"4815dcdafe4f45341ef465a8e4f493cb952eec85b40209ed040c51655f16503b"}
Nov 02 18:01:37.666229 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.656885    4456 scope.go:122] "RemoveContainer" containerID="993c47b2e4ab9a8bec71be49a1d715eb16bb30e2cfae42e995f4ea0dc51b11d2"
Nov 02 18:01:37.710652 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.710615    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"]
Nov 02 18:01:37.735269 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.719240    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-063f48-openshift-default-6f7c89d577-cv77l"]
Nov 02 18:01:37.769131 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.768900    4456 scope.go:122] "RemoveContainer" containerID="993c47b2e4ab9a8bec71be49a1d715eb16bb30e2cfae42e995f4ea0dc51b11d2"
Nov 02 18:01:37.782503 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:01:37.782418    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"993c47b2e4ab9a8bec71be49a1d715eb16bb30e2cfae42e995f4ea0dc51b11d2\": container with ID starting with 993c47b2e4ab9a8bec71be49a1d715eb16bb30e2cfae42e995f4ea0dc51b11d2 not found: ID does not exist" containerID="993c47b2e4ab9a8bec71be49a1d715eb16bb30e2cfae42e995f4ea0dc51b11d2"
Nov 02 18:01:37.782503 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.782466    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"993c47b2e4ab9a8bec71be49a1d715eb16bb30e2cfae42e995f4ea0dc51b11d2"} err="failed to get container status \"993c47b2e4ab9a8bec71be49a1d715eb16bb30e2cfae42e995f4ea0dc51b11d2\": rpc error: code = NotFound desc = could not find container \"993c47b2e4ab9a8bec71be49a1d715eb16bb30e2cfae42e995f4ea0dc51b11d2\": container with ID starting with 993c47b2e4ab9a8bec71be49a1d715eb16bb30e2cfae42e995f4ea0dc51b11d2 not found: ID does not exist"
Nov 02 18:01:37.825643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.825620    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"
Nov 02 18:01:37.919545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.919505    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-maas-api-xks-ca\") pod \"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0\" (UID: \"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0\") "
Nov 02 18:01:37.919754 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.919731    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-maas-api-openshift-ca\") pod \"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0\" (UID: \"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0\") "
Nov 02 18:01:37.919922 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.919907    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-kube-api-access-wn9m7\" (UniqueName: \"kubernetes.io/projected/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-kube-api-access-wn9m7\") pod \"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0\" (UID: \"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0\") "
Nov 02 18:01:37.920536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.920132    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-maas-api-openshift-ca" pod "e367b7a1-9c38-451a-a8dc-e8b8d2698fb0" (UID: "e367b7a1-9c38-451a-a8dc-e8b8d2698fb0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:01:37.926033 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.920919    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:37.926033 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.922134    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-maas-api-xks-ca" pod "e367b7a1-9c38-451a-a8dc-e8b8d2698fb0" (UID: "e367b7a1-9c38-451a-a8dc-e8b8d2698fb0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:01:37.926033 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:37.922899    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-kube-api-access-wn9m7" pod "e367b7a1-9c38-451a-a8dc-e8b8d2698fb0" (UID: "e367b7a1-9c38-451a-a8dc-e8b8d2698fb0"). InnerVolumeSpecName "kube-api-access-wn9m7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:38.038447 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:38.022693    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wn9m7\" (UniqueName: \"kubernetes.io/projected/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-kube-api-access-wn9m7\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:38.038447 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:38.022718    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:38.636437 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:38.636409    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"
Nov 02 18:01:38.637015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:38.636450    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx" event={"ID":"e367b7a1-9c38-451a-a8dc-e8b8d2698fb0","Type":"ContainerDied","Data":"697d8b131e833198d552548be152fe68ff07b10ff277142ebc91e224422bdfec"}
Nov 02 18:01:38.637015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:38.636494    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="697d8b131e833198d552548be152fe68ff07b10ff277142ebc91e224422bdfec"
Nov 02 18:01:38.860240 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:38.860209    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"]
Nov 02 18:01:38.861353 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:38.861326    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e367b7a1-9c38-451a-a8dc-e8b8d2698fb0" containerName="revoke-keys"
Nov 02 18:01:38.861353 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:38.861351    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e367b7a1-9c38-451a-a8dc-e8b8d2698fb0" containerName="revoke-keys"
Nov 02 18:01:38.925841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:38.861390    4456 cpu_manager.go:414] "containerMap: removing container" podUID="ab077054-0cce-44bd-aecc-ee2938d106c9" containerName="istio-proxy"
Nov 02 18:01:38.925841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:38.861397    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ab077054-0cce-44bd-aecc-ee2938d106c9" containerName="istio-proxy"
Nov 02 18:01:38.925841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:38.861533    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ab077054-0cce-44bd-aecc-ee2938d106c9" containerName="istio-proxy"
Nov 02 18:01:38.925841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:38.861552    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e367b7a1-9c38-451a-a8dc-e8b8d2698fb0" containerName="revoke-keys"
Nov 02 18:01:38.966649 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:38.951992    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"]
Nov 02 18:01:38.966649 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:38.952111    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.143104 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.143070    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.143299 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.143123    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.143299 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.143160    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.143299 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.143226    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.143299 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.143265    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.143299 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.143294    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.143581 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.143395    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.143581 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.143427    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.143581 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.143468    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9ms8q\" (UniqueName: \"kubernetes.io/projected/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-kube-api-access-9ms8q\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.209745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.209689    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:39.245270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.245242    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.245465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.245282    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.245465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.245309    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.245465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.245350    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.245465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.245377    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.245465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.245393    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.293098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.245504    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.293098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.245530    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.293098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.245564    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9ms8q\" (UniqueName: \"kubernetes.io/projected/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-kube-api-access-9ms8q\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.293098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.245704    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.293098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.245734    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.293098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.286363    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.293098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.286707    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.293098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.286735    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.293098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.286737    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.293098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.288866    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.347482 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.314726    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.347482 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.314877    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9ms8q\" (UniqueName: \"kubernetes.io/projected/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-kube-api-access-9ms8q\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.347482 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.336676    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"]
Nov 02 18:01:39.347482 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.336897    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt" podUID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerName="istio-proxy" containerID="cri-o://e9384c1e62fcac2aa6d54f5deceeef3155733458debf08d0a878bd911b7b27de" gracePeriod=30
Nov 02 18:01:39.607787 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.607694    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:39.621231 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.621189    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ab077054-0cce-44bd-aecc-ee2938d106c9" path="/var/lib/kubelet/pods/ab077054-0cce-44bd-aecc-ee2938d106c9/volumes"
Nov 02 18:01:39.796616 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:39.796570    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:40.105922 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:40.093622    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:01:40.105922 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:40.093682    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:40.765832 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:40.765772    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"]
Nov 02 18:01:40.846474 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:01:40.831420    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5b0ff069_2a17_489e_ab3d_6d5461bffbf4.slice/crio-bc071adec7a494208311ed9826625b1df3e9551774073ebf93a334bf665d7cb5 WatchSource:0}: Error finding container bc071adec7a494208311ed9826625b1df3e9551774073ebf93a334bf665d7cb5: Status 404 returned error can't find the container with id bc071adec7a494208311ed9826625b1df3e9551774073ebf93a334bf665d7cb5
Nov 02 18:01:40.902213 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:40.902178    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:40.943048 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:40.943007    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:40.943218 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:40.943102    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:40.943218 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:40.943149    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:41.460444 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:41.460412    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:41.604551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:41.604468    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:41.666670 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:41.666635    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" event={"ID":"5b0ff069-2a17-489e-ab3d-6d5461bffbf4","Type":"ContainerStarted","Data":"34675a9bb3e0d78cbe38155c7ebf2cd25bd2500003af0cd5b590e6ab4a83d61a"}
Nov 02 18:01:41.701633 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:41.666692    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" event={"ID":"5b0ff069-2a17-489e-ab3d-6d5461bffbf4","Type":"ContainerStarted","Data":"bc071adec7a494208311ed9826625b1df3e9551774073ebf93a334bf665d7cb5"}
Nov 02 18:01:41.730202 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:41.730147    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podStartSLOduration=3.730132213 podStartE2EDuration="3.730132213s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:38 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:41.726003907 +0000 UTC m=+1918.217067086" watchObservedRunningTime="2026-11-02 18:01:41.730132213 +0000 UTC m=+1918.221195374"
Nov 02 18:01:41.982474 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:41.982437    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:42.599463 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:42.599433    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"]
Nov 02 18:01:42.612253 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:42.608736    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"]
Nov 02 18:01:42.612253 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:42.610537    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 02 18:01:42.612253 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:42.610593    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 02 18:01:42.647845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:42.646967    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:42.724837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:42.707256    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:42.868162 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:42.868076    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:43.624838 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:43.619989    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 02 18:01:43.624838 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:43.620064    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 02 18:01:43.696804 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:43.696767    4456 generic.go:356] "Generic (PLEG): container finished" podID="38cc3415-dd2f-4e7a-91a9-f3740f2cc826" containerID="3ca3b8e733f06e462266217f185ef0da9df2ed08e368a7c11910091f122469d5" exitCode=0
Nov 02 18:01:43.698208 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:43.698178    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n" event={"ID":"38cc3415-dd2f-4e7a-91a9-f3740f2cc826","Type":"ContainerDied","Data":"3ca3b8e733f06e462266217f185ef0da9df2ed08e368a7c11910091f122469d5"}
Nov 02 18:01:44.496843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:44.493516    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:44.623841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:44.620919    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 02 18:01:44.623841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:44.620981    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 02 18:01:45.650034 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:45.631113    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 02 18:01:45.650034 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:45.631172    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 02 18:01:45.739771 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:45.739727    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n" event={"ID":"38cc3415-dd2f-4e7a-91a9-f3740f2cc826","Type":"ContainerDied","Data":"9200dbe1a3f049104ff025fbbb6a25c2154a88fbf44439d968e06ec2cae8072a"}
Nov 02 18:01:45.740010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:45.739868    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9200dbe1a3f049104ff025fbbb6a25c2154a88fbf44439d968e06ec2cae8072a"
Nov 02 18:01:45.742416 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:45.742383    4456 generic.go:356] "Generic (PLEG): container finished" podID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerID="e9384c1e62fcac2aa6d54f5deceeef3155733458debf08d0a878bd911b7b27de" exitCode=0
Nov 02 18:01:45.742565 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:45.742455    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt" event={"ID":"627671f3-ee4e-4ff9-b03f-235489bc356f","Type":"ContainerDied","Data":"e9384c1e62fcac2aa6d54f5deceeef3155733458debf08d0a878bd911b7b27de"}
Nov 02 18:01:45.742565 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:45.742512    4456 scope.go:122] "RemoveContainer" containerID="6f61a792111d21e8b0f4badd27b2c7c04693431f7923f8a8a2d0449f05484e9e"
Nov 02 18:01:45.769502 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:45.755774    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"
Nov 02 18:01:45.858351 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:45.857800    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38cc3415-dd2f-4e7a-91a9-f3740f2cc826-kube-api-access-pzwd9\" (UniqueName: \"kubernetes.io/projected/38cc3415-dd2f-4e7a-91a9-f3740f2cc826-kube-api-access-pzwd9\") pod \"38cc3415-dd2f-4e7a-91a9-f3740f2cc826\" (UID: \"38cc3415-dd2f-4e7a-91a9-f3740f2cc826\") "
Nov 02 18:01:45.858351 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:45.858205    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38cc3415-dd2f-4e7a-91a9-f3740f2cc826-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38cc3415-dd2f-4e7a-91a9-f3740f2cc826-plugins-config-volume\") pod \"38cc3415-dd2f-4e7a-91a9-f3740f2cc826\" (UID: \"38cc3415-dd2f-4e7a-91a9-f3740f2cc826\") "
Nov 02 18:01:45.926111 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:45.926048    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38cc3415-dd2f-4e7a-91a9-f3740f2cc826-plugins-config-volume" pod "38cc3415-dd2f-4e7a-91a9-f3740f2cc826" (UID: "38cc3415-dd2f-4e7a-91a9-f3740f2cc826"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:01:45.961924 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:45.961846    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/38cc3415-dd2f-4e7a-91a9-f3740f2cc826-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:46.645115 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.610102    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 02 18:01:46.645115 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.610172    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 02 18:01:46.674143 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.670497    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.816990    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt" event={"ID":"627671f3-ee4e-4ff9-b03f-235489bc356f","Type":"ContainerDied","Data":"708d50b71df2a3bb0fe9fa5d1c85353d7c1b0d39fd39683dbf0a12a97a845e9e"}
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.817058    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="708d50b71df2a3bb0fe9fa5d1c85353d7c1b0d39fd39683dbf0a12a97a845e9e"
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.817066    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.850313    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38cc3415-dd2f-4e7a-91a9-f3740f2cc826-kube-api-access-pzwd9" pod "38cc3415-dd2f-4e7a-91a9-f3740f2cc826" (UID: "38cc3415-dd2f-4e7a-91a9-f3740f2cc826"). InnerVolumeSpecName "kube-api-access-pzwd9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.850338    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.886051    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-credential-socket\") pod \"627671f3-ee4e-4ff9-b03f-235489bc356f\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") "
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.886208    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-data\") pod \"627671f3-ee4e-4ff9-b03f-235489bc356f\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") "
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.886318    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-token\" (UniqueName: \"kubernetes.io/projected/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-token\") pod \"627671f3-ee4e-4ff9-b03f-235489bc356f\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") "
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.886411    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-workload-certs\") pod \"627671f3-ee4e-4ff9-b03f-235489bc356f\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") "
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.886503    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-podinfo\") pod \"627671f3-ee4e-4ff9-b03f-235489bc356f\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") "
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.886530    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-envoy\") pod \"627671f3-ee4e-4ff9-b03f-235489bc356f\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") "
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.886604    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/627671f3-ee4e-4ff9-b03f-235489bc356f-kube-api-access-l2d6n\" (UniqueName: \"kubernetes.io/projected/627671f3-ee4e-4ff9-b03f-235489bc356f-kube-api-access-l2d6n\") pod \"627671f3-ee4e-4ff9-b03f-235489bc356f\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") "
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.886654    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-workload-socket\") pod \"627671f3-ee4e-4ff9-b03f-235489bc356f\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") "
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.886695    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/627671f3-ee4e-4ff9-b03f-235489bc356f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627671f3-ee4e-4ff9-b03f-235489bc356f-istiod-ca-cert\") pod \"627671f3-ee4e-4ff9-b03f-235489bc356f\" (UID: \"627671f3-ee4e-4ff9-b03f-235489bc356f\") "
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.887477    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pzwd9\" (UniqueName: \"kubernetes.io/projected/38cc3415-dd2f-4e7a-91a9-f3740f2cc826-kube-api-access-pzwd9\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.888101    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-workload-certs" pod "627671f3-ee4e-4ff9-b03f-235489bc356f" (UID: "627671f3-ee4e-4ff9-b03f-235489bc356f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.888745    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-data" pod "627671f3-ee4e-4ff9-b03f-235489bc356f" (UID: "627671f3-ee4e-4ff9-b03f-235489bc356f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.897714    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-workload-socket" pod "627671f3-ee4e-4ff9-b03f-235489bc356f" (UID: "627671f3-ee4e-4ff9-b03f-235489bc356f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.920108    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-envoy" pod "627671f3-ee4e-4ff9-b03f-235489bc356f" (UID: "627671f3-ee4e-4ff9-b03f-235489bc356f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:46.997398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:46.995968    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627671f3-ee4e-4ff9-b03f-235489bc356f-kube-api-access-l2d6n" pod "627671f3-ee4e-4ff9-b03f-235489bc356f" (UID: "627671f3-ee4e-4ff9-b03f-235489bc356f"). InnerVolumeSpecName "kube-api-access-l2d6n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.007067    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.007097    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.007109    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.007124    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l2d6n\" (UniqueName: \"kubernetes.io/projected/627671f3-ee4e-4ff9-b03f-235489bc356f-kube-api-access-l2d6n\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.007139    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.047489    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-token" pod "627671f3-ee4e-4ff9-b03f-235489bc356f" (UID: "627671f3-ee4e-4ff9-b03f-235489bc356f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.101115    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.101197    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.117071    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/627671f3-ee4e-4ff9-b03f-235489bc356f-istiod-ca-cert" pod "627671f3-ee4e-4ff9-b03f-235489bc356f" (UID: "627671f3-ee4e-4ff9-b03f-235489bc356f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.117082    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-credential-socket" pod "627671f3-ee4e-4ff9-b03f-235489bc356f" (UID: "627671f3-ee4e-4ff9-b03f-235489bc356f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.122874    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/627671f3-ee4e-4ff9-b03f-235489bc356f-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.122911    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/627671f3-ee4e-4ff9-b03f-235489bc356f-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.122928    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.133096    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-podinfo" pod "627671f3-ee4e-4ff9-b03f-235489bc356f" (UID: "627671f3-ee4e-4ff9-b03f-235489bc356f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.217867    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"]
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.223036    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-063f48-5f6bbccbc5-b975n"]
Nov 02 18:01:47.347301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.224212    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/627671f3-ee4e-4ff9-b03f-235489bc356f-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:01:47.711018 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.613066    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt" podUID="7b59811f-df47-421a-9e15-d31e4b7bb248" containerName="payload-processing" containerID="cri-o://dc9a939e2490fa79d04eec61eaf6693e4dc48aa0258af2733a23d95511593245" gracePeriod=25
Nov 02 18:01:47.711018 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.613609    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8" podUID="93171ede-4b58-4cdc-9467-f9a6c7574817" containerName="payload-pre-processing" containerID="cri-o://992d2a3e54d4db746448b02f06786266b9c02e56729836a4f100dc885f70cf7f" gracePeriod=25
Nov 02 18:01:47.711018 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.617646    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 02 18:01:47.711018 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.617698    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 02 18:01:47.711018 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.629933    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38cc3415-dd2f-4e7a-91a9-f3740f2cc826" path="/var/lib/kubelet/pods/38cc3415-dd2f-4e7a-91a9-f3740f2cc826/volumes"
Nov 02 18:01:47.928095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.863369    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"
Nov 02 18:01:47.928095 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:01:47.914490    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:01:47.928095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.925583    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"]
Nov 02 18:01:47.930246 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:47.930223    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w5-504de7-openshift-default-58468d8575-qgdwt"]
Nov 02 18:01:48.610839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:48.610468    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused" start-of-body=
Nov 02 18:01:48.610839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:48.610537    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": dial tcp 10.134.0.223:15021: connect: connection refused"
Nov 02 18:01:48.709871 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:48.708347    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:48.849809 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:48.849772    4456 scope.go:122] "RemoveContainer" containerID="3ca3b8e733f06e462266217f185ef0da9df2ed08e368a7c11910091f122469d5"
Nov 02 18:01:48.957551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:48.957519    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt" podUID="7b59811f-df47-421a-9e15-d31e4b7bb248" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:01:49.000969 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:48.957674    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8" podUID="93171ede-4b58-4cdc-9467-f9a6c7574817" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:01:49.122278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:49.121904    4456 scope.go:122] "RemoveContainer" containerID="e9384c1e62fcac2aa6d54f5deceeef3155733458debf08d0a878bd911b7b27de"
Nov 02 18:01:49.375543 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:49.375516    4456 scope.go:122] "RemoveContainer" containerID="959c9a72d603d65a409a8b2c5aafe12f1e188fd50b942870327fa4707b5b135b"
Nov 02 18:01:49.616657 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:49.616623    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="627671f3-ee4e-4ff9-b03f-235489bc356f" path="/var/lib/kubelet/pods/627671f3-ee4e-4ff9-b03f-235489bc356f/volumes"
Nov 02 18:01:50.026778 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:50.026744    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:50.027308 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:50.027286    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:50.028645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:50.028624    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:01:50.474378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:50.474332    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:51.517621 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:51.517586    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"]
Nov 02 18:01:51.613243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:51.518580    4456 cpu_manager.go:414] "containerMap: removing container" podUID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerName="istio-proxy"
Nov 02 18:01:51.613243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:51.518599    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerName="istio-proxy"
Nov 02 18:01:51.613243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:51.518622    4456 cpu_manager.go:414] "containerMap: removing container" podUID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerName="istio-proxy"
Nov 02 18:01:51.613243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:51.518628    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerName="istio-proxy"
Nov 02 18:01:51.613243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:51.518641    4456 cpu_manager.go:414] "containerMap: removing container" podUID="38cc3415-dd2f-4e7a-91a9-f3740f2cc826" containerName="payload-processing"
Nov 02 18:01:51.613243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:51.518646    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38cc3415-dd2f-4e7a-91a9-f3740f2cc826" containerName="payload-processing"
Nov 02 18:01:51.613243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:51.518804    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerName="istio-proxy"
Nov 02 18:01:51.613243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:51.518833    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="38cc3415-dd2f-4e7a-91a9-f3740f2cc826" containerName="payload-processing"
Nov 02 18:01:51.884323 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:51.884238    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:51.901712 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:51.901662    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"]
Nov 02 18:01:52.028764 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.028728    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istiod-ca-cert\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.028764 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.028769    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-token\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.028958    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-credential-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.029055    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-data\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.029091    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8w2nc\" (UniqueName: \"kubernetes.io/projected/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-kube-api-access-8w2nc\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.029235    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-workload-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.029293    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-podinfo\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.029487    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-workload-certs\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.029611    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-envoy\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.131577    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-data\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.131625    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8w2nc\" (UniqueName: \"kubernetes.io/projected/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-kube-api-access-8w2nc\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.131692    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-workload-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.131728    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-podinfo\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.131828    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-workload-certs\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.131892    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-envoy\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.131951    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istiod-ca-cert\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.131970    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-token\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.132016    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-credential-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.132359    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-credential-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.132515    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-data\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.132958    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-workload-socket\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.133302    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-workload-certs\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.146172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.133631    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istiod-ca-cert\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.304967 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.304929    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:52.317014 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.308388    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-podinfo\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.341340 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.341286    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8w2nc\" (UniqueName: \"kubernetes.io/projected/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-kube-api-access-8w2nc\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.341533 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.341511    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-envoy\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.342629 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.341906    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-token\") pod \"e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") " pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.410375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.410278    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:52.522491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:52.522447    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:53.639867 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:53.639794    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:53.640205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:53.639919    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:53.640205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:53.639984    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:53.643038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:53.641139    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"]
Nov 02 18:01:53.901180 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:53.901072    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"]
Nov 02 18:01:53.929213 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:53.908751    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="627671f3-ee4e-4ff9-b03f-235489bc356f" containerName="istio-proxy"
Nov 02 18:01:54.099371 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.099336    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:54.142076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.142044    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"]
Nov 02 18:01:54.142265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.142160    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.250921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.192171    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.250921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.192303    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xn778\" (UniqueName: \"kubernetes.io/projected/38977edb-8612-48c6-a6c6-1714a50a9e4f-kube-api-access-xn778\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.250921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.192346    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38977edb-8612-48c6-a6c6-1714a50a9e4f-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.250921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.192387    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.250921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.192463    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.250921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.192550    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.250921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.192595    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.250921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.192668    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.250921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.192809    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.265309 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.250930    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"]
Nov 02 18:01:54.265309 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.261354    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.297471 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.297430    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.297782 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.297755    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g9h5l\" (UniqueName: \"kubernetes.io/projected/806cc8ce-f608-4975-94dd-5edade11fc8c-kube-api-access-g9h5l\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.298065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.298048    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.298184 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.298156    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.298356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.298321    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xn778\" (UniqueName: \"kubernetes.io/projected/38977edb-8612-48c6-a6c6-1714a50a9e4f-kube-api-access-xn778\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.298532 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.298517    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38977edb-8612-48c6-a6c6-1714a50a9e4f-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.298697 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.298676    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.298861 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.298846    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.299021 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.299001    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.299178 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.299156    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/806cc8ce-f608-4975-94dd-5edade11fc8c-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.299389 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.299360    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38977edb-8612-48c6-a6c6-1714a50a9e4f-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.299599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.299574    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.302610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.300220    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.302610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.300323    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.302610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.300376    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.302610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.300515    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.302610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.300607    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.302610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.300636    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.302610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.300694    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.302610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.300746    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.302610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.301658    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.302610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.302094    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.302610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.302509    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.332744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.310433    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.332744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.314296    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"]
Nov 02 18:01:54.332744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.314349    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" event={"ID":"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3","Type":"ContainerStarted","Data":"a6ef42ab6840fc568f9b0249306f579e85614adaade0478bce59dcb673d5d337"}
Nov 02 18:01:54.332744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.314385    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"]
Nov 02 18:01:54.332744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.314417    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"]
Nov 02 18:01:54.332744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.314519    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.332744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.319679    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.332744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.325741    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xn778\" (UniqueName: \"kubernetes.io/projected/38977edb-8612-48c6-a6c6-1714a50a9e4f-kube-api-access-xn778\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.332744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.326743    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.402886 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.402777    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.403081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.402894    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.403081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.402940    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.403081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.402977    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.403081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.403006    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.403081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.403039    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.403081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.403073    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.415609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.403173    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.415609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.403231    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g9h5l\" (UniqueName: \"kubernetes.io/projected/806cc8ce-f608-4975-94dd-5edade11fc8c-kube-api-access-g9h5l\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.415609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.403284    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.415609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.403322    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.415609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.403410    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.415609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.403442    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.415609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.403484    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/806cc8ce-f608-4975-94dd-5edade11fc8c-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.415609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.403602    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.415609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.403733    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.415609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.403758    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g75j4\" (UniqueName: \"kubernetes.io/projected/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-kube-api-access-g75j4\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.415609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.403873    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.427580 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.420519    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.427580 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.423890    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.427580 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.423890    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.427580 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.424343    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.427580 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.424422    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/806cc8ce-f608-4975-94dd-5edade11fc8c-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.427580 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.424654    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.433466 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.433376    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.437245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.437220    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.447471 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.447443    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g9h5l\" (UniqueName: \"kubernetes.io/projected/806cc8ce-f608-4975-94dd-5edade11fc8c-kube-api-access-g9h5l\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.507869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.505592    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.507869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.505644    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g75j4\" (UniqueName: \"kubernetes.io/projected/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-kube-api-access-g75j4\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.507869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.505699    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.507869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.505780    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.507869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.506381    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.508306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.508085    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.508306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.508136    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.508306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.508170    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.508306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.508283    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.508534 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.508369    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.508870 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.508848    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.509043 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.509020    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.524664 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.509095    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.527201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.527152    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.538726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.527689    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.538726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.531269    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g75j4\" (UniqueName: \"kubernetes.io/projected/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-kube-api-access-g75j4\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.538726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.531541    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.549110 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.549082    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:54.552216 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.552191    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:54.615216 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.615017    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:54.697223 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:54.696959    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:55.032326 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:01:55.021480    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod38977edb_8612_48c6_a6c6_1714a50a9e4f.slice/crio-74225bd3ac6241676ac73b0acba4384e47d913bbf76c3afc6d0e20827d4fe06b WatchSource:0}: Error finding container 74225bd3ac6241676ac73b0acba4384e47d913bbf76c3afc6d0e20827d4fe06b: Status 404 returned error can't find the container with id 74225bd3ac6241676ac73b0acba4384e47d913bbf76c3afc6d0e20827d4fe06b
Nov 02 18:01:55.032326 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.025612    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"]
Nov 02 18:01:55.032326 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.028022    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" event={"ID":"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3","Type":"ContainerStarted","Data":"8f711850377e64150821c64844dc9b4f619291b996bda5ba3ee277096962757c"}
Nov 02 18:01:55.062865 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.062801    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:55.063020 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.062919    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:55.063020 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.062967    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:55.092746 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.092713    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:01:55.092969 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.092763    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:55.131308 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.131245    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podStartSLOduration=4.131223869 podStartE2EDuration="4.131223869s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:55.079354663 +0000 UTC m=+1931.570417825" watchObservedRunningTime="2026-11-02 18:01:55.131223869 +0000 UTC m=+1931.622287033"
Nov 02 18:01:55.142744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.142262    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"]
Nov 02 18:01:55.171153 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:01:55.171128    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod5e54b0f0_5c3c_4441_a2a6_b452b9715e2e.slice/crio-2300f223a1d1ec0b2987b8462270cde83cca39409c0ffbb2273e03ea9a9f7274 WatchSource:0}: Error finding container 2300f223a1d1ec0b2987b8462270cde83cca39409c0ffbb2273e03ea9a9f7274: Status 404 returned error can't find the container with id 2300f223a1d1ec0b2987b8462270cde83cca39409c0ffbb2273e03ea9a9f7274
Nov 02 18:01:55.174799 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.174765    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:55.174909 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.174859    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:55.174909 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.174893    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:55.242503 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.197383    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"]
Nov 02 18:01:55.242503 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.228926    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:55.242503 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.229023    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:55.242503 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.229075    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:01:55.422031 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.420260    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:01:55.431945 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.423297    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 02 18:01:55.431945 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.423368    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 02 18:01:55.941514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:55.941481    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:56.174222 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.174172    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" event={"ID":"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e","Type":"ContainerStarted","Data":"7d77b8d6127469706b8666ac8056de7ddc308b1c2fb37e9a35ec932c0e735f54"}
Nov 02 18:01:56.174513 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.174478    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" event={"ID":"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e","Type":"ContainerStarted","Data":"2300f223a1d1ec0b2987b8462270cde83cca39409c0ffbb2273e03ea9a9f7274"}
Nov 02 18:01:56.196800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.195157    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" event={"ID":"38977edb-8612-48c6-a6c6-1714a50a9e4f","Type":"ContainerStarted","Data":"95adba9c28b54d3f0298de1dec2292a9810fd828ad712b762f15dc288bbab452"}
Nov 02 18:01:56.196800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.195223    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" event={"ID":"38977edb-8612-48c6-a6c6-1714a50a9e4f","Type":"ContainerStarted","Data":"74225bd3ac6241676ac73b0acba4384e47d913bbf76c3afc6d0e20827d4fe06b"}
Nov 02 18:01:56.217851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.216413    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" event={"ID":"806cc8ce-f608-4975-94dd-5edade11fc8c","Type":"ContainerStarted","Data":"7ea8d9afddf0f6e4f043e8c148239bb0b12e1d9cf9c2338a03073a647dc99771"}
Nov 02 18:01:56.217851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.216476    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" event={"ID":"806cc8ce-f608-4975-94dd-5edade11fc8c","Type":"ContainerStarted","Data":"55c5ffef20baf90e86b8e6c4709e524db068d7d1b2bad7ab617d4d741b856797"}
Nov 02 18:01:56.243453 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.240574    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podStartSLOduration=3.240554346 podStartE2EDuration="3.240554346s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:56.225526928 +0000 UTC m=+1932.716590105" watchObservedRunningTime="2026-11-02 18:01:56.240554346 +0000 UTC m=+1932.731617508"
Nov 02 18:01:56.315886 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.315802    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podStartSLOduration=3.315780011 podStartE2EDuration="3.315780011s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:56.276089438 +0000 UTC m=+1932.767152603" watchObservedRunningTime="2026-11-02 18:01:56.315780011 +0000 UTC m=+1932.806843174"
Nov 02 18:01:56.361784 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.360942    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podStartSLOduration=3.360922058 podStartE2EDuration="3.360922058s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:01:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:01:56.336654339 +0000 UTC m=+1932.827717528" watchObservedRunningTime="2026-11-02 18:01:56.360922058 +0000 UTC m=+1932.851985221"
Nov 02 18:01:56.441489 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.441447    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 02 18:01:56.453523 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.441532    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 02 18:01:56.552873 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.552414    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:01:56.560010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.553914    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 02 18:01:56.560010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.553958    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 02 18:01:56.622941 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.622604    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:01:56.622941 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.622716    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:01:56.622941 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.622757    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 02 18:01:56.701709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.699775    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:01:56.701709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.699929    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 02 18:01:56.701709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:56.699972    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 02 18:01:57.411883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:57.411748    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 02 18:01:57.412643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:57.412605    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 02 18:01:57.572530 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:57.571088    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 02 18:01:57.572530 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:57.571147    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 02 18:01:57.664075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:57.663970    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:01:57.666079 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:57.664272    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 02 18:01:57.718972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:57.718926    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 02 18:01:57.719283 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:57.719254    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 02 18:01:57.798986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:57.762109    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:01:57.866024 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:57.849912    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 18:01:57.973842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:57.973215    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:57.999047 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:57.983367    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:58.140843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.139913    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:58.140843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.140206    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 18:01:58.155550 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.142383    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:58.308901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.307916    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:58.399839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.398149    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:58.424843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.417508    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 02 18:01:58.424843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.417583    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 02 18:01:58.556876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.553508    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 02 18:01:58.556876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.553566    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 02 18:01:58.599892 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.588027    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:58.630842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.618516    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:01:58.630842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.618573    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 02 18:01:58.705840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.705282    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 02 18:01:58.705840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.705342    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 02 18:01:58.982044 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.962410    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8" podUID="93171ede-4b58-4cdc-9467-f9a6c7574817" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:01:58.982044 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.963335    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt" podUID="7b59811f-df47-421a-9e15-d31e4b7bb248" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:01:58.982044 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.973427    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:58.982044 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:58.973804    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 18:01:59.164901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:59.147936    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:59.164901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:59.148359    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 18:01:59.164901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:59.148993    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 18:01:59.164901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:59.150684    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:59.164901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:59.151172    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 18:01:59.316889 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:59.309004    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:01:59.366840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:59.318406    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:01:59.445843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:59.431163    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 02 18:01:59.445843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:59.431225    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 02 18:02:00.002923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:59.528302    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 18:02:00.002923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:59.528869    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 18:02:00.002923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:59.620582    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 02 18:02:00.002923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:59.620640    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 02 18:02:00.002923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:59.767249    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:00.002923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:01:59.973577    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:00.002923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.000386    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:00.002923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.001525    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:02:00.002923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.001568    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 02 18:02:00.002923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.002042    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 02 18:02:00.002923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.002092    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 02 18:02:00.149952 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.135946    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:02:00.185847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.168068    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 18:02:00.209328 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.194984    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 18:02:00.292117 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.280906    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 18:02:00.366559 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.306888    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 18:02:00.366559 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.337941    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:00.366559 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.339592    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:02:00.366559 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.345988    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Nov 02 18:02:00.465842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.465167    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 02 18:02:00.465842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.465227    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 02 18:02:00.539839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.530574    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 18:02:00.591993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.564736    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 02 18:02:00.591993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.569199    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 02 18:02:00.631882 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.630937    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:00.882414 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.873703    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 02 18:02:00.882414 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.873800    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 02 18:02:00.943275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.905503    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:02:00.943275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:00.905574    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 02 18:02:01.009085 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.009017    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 18:02:01.042077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.033202    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 18:02:01.090031 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.049420    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 18:02:01.090031 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.049765    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 18:02:01.309843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.302434    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:01.490024 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.489955    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 18:02:01.583982 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.514650    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Nov 02 18:02:01.583982 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.524635    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:01.718841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.451050    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:01.795881 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.773542    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 02 18:02:01.901971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.797358    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 02 18:02:01.902479 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.902441    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 02 18:02:01.903261 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.815547    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 02 18:02:01.906921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.906882    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 02 18:02:01.907118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.815613    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:02:01.910935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:01.907258    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 02 18:02:02.144841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.095034    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 18:02:02.144841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.117794    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:02.144841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.144278    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:02.148009 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.147970    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:02.295445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.295398    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:02:02.364841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.335411    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 18:02:02.364841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.341021    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:02:02.364841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.341074    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:02.364841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.345058    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 18:02:02.364841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.345149    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 18:02:02.364841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.352592    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 02 18:02:02.364841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.352663    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 02 18:02:02.443840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.418335    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused" start-of-body=
Nov 02 18:02:02.443840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.418393    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": dial tcp 10.134.0.224:15021: connect: connection refused"
Nov 02 18:02:02.485802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.453955    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:02.554746 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.554651    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 02 18:02:02.555445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.555415    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 02 18:02:02.627843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.617588    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 18:02:02.627843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.617975    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 18:02:02.627843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.618625    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:02:02.661227 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.631157    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:02:02.671870 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.670348    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 02 18:02:02.691857 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.683967    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:02.797854 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.796954    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 02 18:02:02.797854 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:02.797013    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 02 18:02:03.157773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:03.156541    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:03.157773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:03.157154    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:03.157773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:03.157501    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 18:02:03.159160 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:03.158338    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 18:02:03.352355 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:03.328179    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:03.374468 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:03.374434    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 18:02:03.375109 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:03.375073    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 18:02:03.678564 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:03.608665    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 18:02:03.678564 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:03.636095    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 18:02:03.678564 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:03.636405    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 18:02:03.708117 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:03.680023    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:03.999899 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:03.885664    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:04.185845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.149641    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:04.469036 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.203159    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 18:02:04.469565 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.204034    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 02 18:02:04.469739 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.469711    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 02 18:02:04.469904 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.294770    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:04.470020 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.385238    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 18:02:04.477993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.431295    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:04.477993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.218288    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:04.610845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.596936    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" podUID="22b556c1-112b-49d4-932c-0143e912a54e" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.186:50052: i/o timeout"
Nov 02 18:02:04.653536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.652526    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 02 18:02:04.653536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.652592    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 02 18:02:04.653536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.653359    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:02:04.653536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.653400    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 02 18:02:04.730149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.716951    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused" start-of-body=
Nov 02 18:02:04.730149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.717026    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": dial tcp 10.134.0.226:15021: connect: connection refused"
Nov 02 18:02:04.730149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.726394    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused" start-of-body=
Nov 02 18:02:04.730149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.726448    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": dial tcp 10.134.0.227:15021: connect: connection refused"
Nov 02 18:02:04.792852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.786398    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:04.834129 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.812765    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 18:02:04.834129 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.814265    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 18:02:04.835989 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:02:04.827298    4456 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.218s"
Nov 02 18:02:04.881891 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.879870    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused" start-of-body=
Nov 02 18:02:04.881891 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.879926    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": dial tcp 10.134.0.225:15021: connect: connection refused"
Nov 02 18:02:04.895697 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.895670    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:02:04.896144 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.896117    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:02:04.897365 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.897345    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:02:04.903568 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.903538    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 18:02:04.905138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.905117    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 18:02:04.921924 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.921889    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:02:04.922614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.922590    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 18:02:04.947141 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.947111    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 18:02:04.956016 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.955988    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 18:02:05.015677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.973040    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 18:02:05.015677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:04.998617    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 18:02:05.362478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:05.360404    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:05.559896 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:05.559862    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:02:05.560492 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:05.560470    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:02:05.561437 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:05.561419    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:02:05.626517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:05.625198    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:02:05.626517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:05.626368    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:02:05.757597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:05.627194    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:02:05.757597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:05.706722    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:02:05.757597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:05.708381    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:02:05.757597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:05.709419    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:02:07.243435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:07.243405    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:09.195653 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:02:09.133850    4456 logging.go:55] [core] [Channel #5310 SubChannel #5312]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.162:50052", ServerName: "10.134.0.162:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.162:50052: connect: connection refused"
Nov 02 18:02:09.195653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:09.135763    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:09.195653 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:02:09.169731    4456 logging.go:55] [core] [Channel #5309 SubChannel #5311]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.161:50052", ServerName: "10.134.0.161:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.161:50052: connect: connection refused"
Nov 02 18:02:09.273368 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:09.242718    4456 generic.go:356] "Generic (PLEG): container finished" podID="7b59811f-df47-421a-9e15-d31e4b7bb248" containerID="dc9a939e2490fa79d04eec61eaf6693e4dc48aa0258af2733a23d95511593245" exitCode=0
Nov 02 18:02:09.273368 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:09.242794    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt" event={"ID":"7b59811f-df47-421a-9e15-d31e4b7bb248","Type":"ContainerDied","Data":"dc9a939e2490fa79d04eec61eaf6693e4dc48aa0258af2733a23d95511593245"}
Nov 02 18:02:09.273368 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:09.244770    4456 generic.go:356] "Generic (PLEG): container finished" podID="93171ede-4b58-4cdc-9467-f9a6c7574817" containerID="992d2a3e54d4db746448b02f06786266b9c02e56729836a4f100dc885f70cf7f" exitCode=0
Nov 02 18:02:09.273368 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:09.244843    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8" event={"ID":"93171ede-4b58-4cdc-9467-f9a6c7574817","Type":"ContainerDied","Data":"992d2a3e54d4db746448b02f06786266b9c02e56729836a4f100dc885f70cf7f"}
Nov 02 18:02:09.956797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:09.956751    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt" podUID="7b59811f-df47-421a-9e15-d31e4b7bb248" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.162:50052\" within 1s: context deadline exceeded"
Nov 02 18:02:09.957076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:09.956751    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8" podUID="93171ede-4b58-4cdc-9467-f9a6c7574817" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.161:50052\" within 1s: context deadline exceeded"
Nov 02 18:02:09.957076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:09.956938    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"
Nov 02 18:02:09.957076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:09.957018    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"
Nov 02 18:02:10.091966 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:10.091933    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:02:10.092177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:10.092001    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:02:10.864089 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:10.864056    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.163296    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.364403    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.404571    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.404585    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt" event={"ID":"7b59811f-df47-421a-9e15-d31e4b7bb248","Type":"ContainerDied","Data":"1c08c37f7f317e0f444b9f9b3c9fa7c2d2443f9cb7a70904fedd5f65a6f659ae"}
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.404655    4456 scope.go:122] "RemoveContainer" containerID="dc9a939e2490fa79d04eec61eaf6693e4dc48aa0258af2733a23d95511593245"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.406345    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8" event={"ID":"93171ede-4b58-4cdc-9467-f9a6c7574817","Type":"ContainerDied","Data":"48fe516f263c0291ee20c126cba079fd01200ccecea8f7a1fc036af4e5eb8568"}
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.406384    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="48fe516f263c0291ee20c126cba079fd01200ccecea8f7a1fc036af4e5eb8568"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.408289    4456 generic.go:356] "Generic (PLEG): container finished" podID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerID="a602b0afbc3a3601e8c878664a9fa511e8e88746ceb9ca1f1a2a019ef2cff058" exitCode=0
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.408362    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" event={"ID":"dbd34da0-9b90-458d-9581-ed9caf3daa11","Type":"ContainerDied","Data":"a602b0afbc3a3601e8c878664a9fa511e8e88746ceb9ca1f1a2a019ef2cff058"}
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.408885    4456 scope.go:122] "RemoveContainer" containerID="a602b0afbc3a3601e8c878664a9fa511e8e88746ceb9ca1f1a2a019ef2cff058"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.430564    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.430637    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.430672    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.466764    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7b59811f-df47-421a-9e15-d31e4b7bb248-kube-api-access-l4tcm\" (UniqueName: \"kubernetes.io/projected/7b59811f-df47-421a-9e15-d31e4b7bb248-kube-api-access-l4tcm\") pod \"7b59811f-df47-421a-9e15-d31e4b7bb248\" (UID: \"7b59811f-df47-421a-9e15-d31e4b7bb248\") "
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.466922    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7b59811f-df47-421a-9e15-d31e4b7bb248-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b59811f-df47-421a-9e15-d31e4b7bb248-plugins-config-volume\") pod \"7b59811f-df47-421a-9e15-d31e4b7bb248\" (UID: \"7b59811f-df47-421a-9e15-d31e4b7bb248\") "
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.494105    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.540417    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"]
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.541328    4456 cpu_manager.go:414] "containerMap: removing container" podUID="7b59811f-df47-421a-9e15-d31e4b7bb248" containerName="payload-processing"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.541344    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7b59811f-df47-421a-9e15-d31e4b7bb248" containerName="payload-processing"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.541383    4456 cpu_manager.go:414] "containerMap: removing container" podUID="93171ede-4b58-4cdc-9467-f9a6c7574817" containerName="payload-pre-processing"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.541388    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="93171ede-4b58-4cdc-9467-f9a6c7574817" containerName="payload-pre-processing"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.541537    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7b59811f-df47-421a-9e15-d31e4b7bb248" containerName="payload-processing"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.541554    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="93171ede-4b58-4cdc-9467-f9a6c7574817" containerName="payload-pre-processing"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.560859    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7b59811f-df47-421a-9e15-d31e4b7bb248-plugins-config-volume" pod "7b59811f-df47-421a-9e15-d31e4b7bb248" (UID: "7b59811f-df47-421a-9e15-d31e4b7bb248"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.568577    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/93171ede-4b58-4cdc-9467-f9a6c7574817-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93171ede-4b58-4cdc-9467-f9a6c7574817-plugins-config-volume\") pod \"93171ede-4b58-4cdc-9467-f9a6c7574817\" (UID: \"93171ede-4b58-4cdc-9467-f9a6c7574817\") "
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.568684    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/93171ede-4b58-4cdc-9467-f9a6c7574817-kube-api-access-dd4rk\" (UniqueName: \"kubernetes.io/projected/93171ede-4b58-4cdc-9467-f9a6c7574817-kube-api-access-dd4rk\") pod \"93171ede-4b58-4cdc-9467-f9a6c7574817\" (UID: \"93171ede-4b58-4cdc-9467-f9a6c7574817\") "
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.569354    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7b59811f-df47-421a-9e15-d31e4b7bb248-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.692632    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/93171ede-4b58-4cdc-9467-f9a6c7574817-plugins-config-volume" pod "93171ede-4b58-4cdc-9467-f9a6c7574817" (UID: "93171ede-4b58-4cdc-9467-f9a6c7574817"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.773262    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/93171ede-4b58-4cdc-9467-f9a6c7574817-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.835115    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.874309    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.874447    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.874510    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.874571    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.874729    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-wx6r2\" (UniqueName: \"kubernetes.io/projected/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-kube-api-access-wx6r2\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.874778    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.874897    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.874931    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.874948    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.976887    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.976961    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.976990    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.977028    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.977183    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.977271    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.977355    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.977527    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-wx6r2\" (UniqueName: \"kubernetes.io/projected/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-kube-api-access-wx6r2\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:13.977598    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.169516    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-workload-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.170243    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-workload-certs\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.170323    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-credential-socket\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.173005    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/93171ede-4b58-4cdc-9467-f9a6c7574817-kube-api-access-dd4rk" pod "93171ede-4b58-4cdc-9467-f9a6c7574817" (UID: "93171ede-4b58-4cdc-9467-f9a6c7574817"). InnerVolumeSpecName "kube-api-access-dd4rk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.173055    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-data\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.173234    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istiod-ca-cert\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.182033    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dd4rk\" (UniqueName: \"kubernetes.io/projected/93171ede-4b58-4cdc-9467-f9a6c7574817-kube-api-access-dd4rk\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.207429    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7b59811f-df47-421a-9e15-d31e4b7bb248-kube-api-access-l4tcm" pod "7b59811f-df47-421a-9e15-d31e4b7bb248" (UID: "7b59811f-df47-421a-9e15-d31e4b7bb248"). InnerVolumeSpecName "kube-api-access-l4tcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.238348    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-podinfo\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.238665    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-wx6r2\" (UniqueName: \"kubernetes.io/projected/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-kube-api-access-wx6r2\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.239146    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-token\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.239269    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-envoy\") pod \"e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") " pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.284561    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l4tcm\" (UniqueName: \"kubernetes.io/projected/7b59811f-df47-421a-9e15-d31e4b7bb248-kube-api-access-l4tcm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.419596    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.507995    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.553246    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"]
Nov 02 18:02:14.741181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.568962    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-b913b2-6755d64865-8mmkt"]
Nov 02 18:02:15.263010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.837262    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"]
Nov 02 18:02:15.263010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.842060    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-b913b2-59969dbd75-rqtc8"]
Nov 02 18:02:15.263010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:14.876620    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:15.263010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:15.179091    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"]
Nov 02 18:02:15.439077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:15.435440    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" event={"ID":"dbd34da0-9b90-458d-9581-ed9caf3daa11","Type":"ContainerStarted","Data":"4cb6844084f3272f4de73238806060cd7601c7e4e7b1a56e862c1e414d1e1d67"}
Nov 02 18:02:15.629367 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:15.629287    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7b59811f-df47-421a-9e15-d31e4b7bb248" path="/var/lib/kubelet/pods/7b59811f-df47-421a-9e15-d31e4b7bb248/volumes"
Nov 02 18:02:15.638869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:15.638800    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="93171ede-4b58-4cdc-9467-f9a6c7574817" path="/var/lib/kubelet/pods/93171ede-4b58-4cdc-9467-f9a6c7574817/volumes"
Nov 02 18:02:15.718306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:15.682172    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"]
Nov 02 18:02:15.718306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:15.685586    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:15.718306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:15.685666    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:15.718306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:15.685707    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:16.053300 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:16.053265    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 18:02:16.085379 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:16.085339    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 18:02:16.086325 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:16.086292    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 02 18:02:16.086479 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:16.086345    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 02 18:02:16.460353 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:16.460300    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" event={"ID":"8cba3353-fa51-41d3-acb4-a3cfbaae1daa","Type":"ContainerStarted","Data":"6f672ff50b415c9f5d877d39fc43cbb4be0f8cf802d9f572dcc1a950321d5ed3"}
Nov 02 18:02:16.460555 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:16.460394    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" event={"ID":"8cba3353-fa51-41d3-acb4-a3cfbaae1daa","Type":"ContainerStarted","Data":"8d9e2ce44f4c31d7512dd74da1f7a331b19098c6711368e074e5c1729d3cf186"}
Nov 02 18:02:16.509733 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:16.508494    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:16.634052 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:16.510596    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 02 18:02:16.634052 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:16.510648    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 02 18:02:16.634052 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:16.513908    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" podStartSLOduration=8.51389273 podStartE2EDuration="8.51389273s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:08 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:16.509063893 +0000 UTC m=+1953.000127066" watchObservedRunningTime="2026-11-02 18:02:16.51389273 +0000 UTC m=+1953.004955893"
Nov 02 18:02:16.673868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:16.673434    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:17.089330 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:17.089292    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 02 18:02:17.113433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:17.099927    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 02 18:02:17.330002 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:17.256033    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:17.513842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:17.511294    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 02 18:02:17.513842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:17.511362    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 02 18:02:17.968206 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:02:17.968128    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:02:18.114328 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:18.099745    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 02 18:02:18.114328 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:18.099802    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 02 18:02:18.513855 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:18.511450    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 02 18:02:18.513855 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:18.511511    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 02 18:02:18.711468 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:18.711429    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:19.107134 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:19.106575    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 02 18:02:19.107134 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:19.106652    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 02 18:02:19.539803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:19.539287    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 02 18:02:19.539803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:19.539370    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 02 18:02:20.119840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:20.111334    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 02 18:02:20.119840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:20.111411    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 02 18:02:20.512706 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:20.512226    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 02 18:02:20.512706 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:20.512302    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 02 18:02:20.552186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:20.552032    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:21.102839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:21.102242    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 02 18:02:21.102839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:21.102329    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 02 18:02:21.535315 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:21.517213    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 02 18:02:21.535315 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:21.517287    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 02 18:02:21.971925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:21.970598    4456 generic.go:356] "Generic (PLEG): container finished" podID="999df778-cd9b-4b41-852d-4661a7326483" containerID="1f92044f46d783f762d4c677bea78e3b2f2e6ab30bddaea987c9b15454088386" exitCode=0
Nov 02 18:02:21.971925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:21.970894    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" event={"ID":"999df778-cd9b-4b41-852d-4661a7326483","Type":"ContainerDied","Data":"1f92044f46d783f762d4c677bea78e3b2f2e6ab30bddaea987c9b15454088386"}
Nov 02 18:02:21.971925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:21.971578    4456 scope.go:122] "RemoveContainer" containerID="1f92044f46d783f762d4c677bea78e3b2f2e6ab30bddaea987c9b15454088386"
Nov 02 18:02:21.975297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:21.974962    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:21.975297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:21.975067    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:21.975297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:21.975119    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:22.175002 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:22.174960    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused" start-of-body=
Nov 02 18:02:22.175220 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:22.175027    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": dial tcp 10.134.0.199:15021: connect: connection refused"
Nov 02 18:02:22.488945 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:22.488903    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:22.511483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:22.511451    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused" start-of-body=
Nov 02 18:02:22.511905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:22.511876    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": dial tcp 10.134.0.228:15021: connect: connection refused"
Nov 02 18:02:22.985202 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:22.985159    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" event={"ID":"999df778-cd9b-4b41-852d-4661a7326483","Type":"ContainerStarted","Data":"c492b110991299865769d96c519b16a0cdfd155200b1d4b35cdb7daa5de888f7"}
Nov 02 18:02:23.091098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:23.091063    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 18:02:23.092397 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:23.092372    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 18:02:23.514281 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:23.514249    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:23.514801 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:23.514783    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:23.515751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:23.515726    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:02:23.702840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:23.661069    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 18:02:23.702840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:23.661551    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 02 18:02:23.702840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:23.661585    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 02 18:02:24.050326 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:24.050289    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 18:02:24.442179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:24.442141    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:24.661581 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:24.661545    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 02 18:02:24.661772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:24.661594    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 02 18:02:25.661604 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:25.661569    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 02 18:02:25.662928 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:25.661632    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 02 18:02:26.305484 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:26.305416    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:26.661656 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:26.661571    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 02 18:02:26.661656 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:26.661634    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 02 18:02:27.661766 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:27.661730    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 02 18:02:27.662260 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:27.661797    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 02 18:02:28.358852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:28.353896    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:28.662148 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:28.662071    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused" start-of-body=
Nov 02 18:02:28.662148 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:28.662125    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": dial tcp 10.134.0.193:15021: connect: connection refused"
Nov 02 18:02:28.793386 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:28.793333    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:29.667180 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:29.667147    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 18:02:29.668310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:29.668285    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 18:02:30.007264 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:30.007228    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:30.039939 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:30.039874    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"]
Nov 02 18:02:30.052007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:30.051974    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ed264d03-vc2hg"]
Nov 02 18:02:30.215867 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:30.215831    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:30.385088 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:30.384999    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:31.619302 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:31.619270    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a75b25ed-ab5b-40e4-8305-59048777fdac" path="/var/lib/kubelet/pods/a75b25ed-ab5b-40e4-8305-59048777fdac/volumes"
Nov 02 18:02:32.152125 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:32.152087    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:32.316234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:32.316191    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:34.125508 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:34.125461    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:35.670119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:35.670081    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:36.491890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:36.481611    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:36.714302 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:36.710840    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:36.756099 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:36.756006    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"]
Nov 02 18:02:36.761614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:36.761586    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-b913b2-c6bgx"]
Nov 02 18:02:36.867786 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:36.867749    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:37.239438 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:37.239408    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:37.623367 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:37.623276    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e367b7a1-9c38-451a-a8dc-e8b8d2698fb0" path="/var/lib/kubelet/pods/e367b7a1-9c38-451a-a8dc-e8b8d2698fb0/volumes"
Nov 02 18:02:38.787842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:38.782638    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:40.711393 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:40.707999    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:42.627842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:42.611319    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:44.484008 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:44.483919    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:44.658219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:44.658188    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:44.931898 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:44.931222    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"]
Nov 02 18:02:44.938415 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:44.938016    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4" podUID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" containerName="istio-proxy" containerID="cri-o://6e6b3298b09594bcb799769b170054e6e96452903ad9152cb54b29e190aafdb3" gracePeriod=30
Nov 02 18:02:44.969997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:44.969520    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:45.816352 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:45.816311    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:46.174523 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:02:46.174438    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:02:46.527848 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:46.527391    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:46.938123 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:46.935353    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:47.115730 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:47.115692    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:48.376970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:48.376897    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:50.182797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.182683    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:50.417107 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.415860    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 18:02:50.503327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.503290    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"]
Nov 02 18:02:50.504721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.504693    4456 cpu_manager.go:414] "containerMap: removing container" podUID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" containerName="istio-proxy"
Nov 02 18:02:50.504721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.504722    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" containerName="istio-proxy"
Nov 02 18:02:50.505093 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.504742    4456 cpu_manager.go:414] "containerMap: removing container" podUID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" containerName="istio-proxy"
Nov 02 18:02:50.505093 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.504750    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" containerName="istio-proxy"
Nov 02 18:02:50.505093 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.505076    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" containerName="istio-proxy"
Nov 02 18:02:50.512642 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.512618    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.519879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.519427    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-workload-socket\") pod \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") "
Nov 02 18:02:50.519879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.519528    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-data\") pod \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") "
Nov 02 18:02:50.519879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.519559    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-workload-certs\") pod \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") "
Nov 02 18:02:50.519879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.519609    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-envoy\") pod \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") "
Nov 02 18:02:50.519879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.519643    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-podinfo\") pod \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") "
Nov 02 18:02:50.521173 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.520063    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istiod-ca-cert\") pod \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") "
Nov 02 18:02:50.521173 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.520113    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-credential-socket\") pod \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") "
Nov 02 18:02:50.521173 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.520155    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-kube-api-access-gszm5\" (UniqueName: \"kubernetes.io/projected/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-kube-api-access-gszm5\") pod \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") "
Nov 02 18:02:50.521173 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.520266    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-token\" (UniqueName: \"kubernetes.io/projected/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-token\") pod \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\" (UID: \"65156215-be49-4a1b-a7fd-2b1eb2f3f13d\") "
Nov 02 18:02:50.523153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.521460    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-workload-certs" pod "65156215-be49-4a1b-a7fd-2b1eb2f3f13d" (UID: "65156215-be49-4a1b-a7fd-2b1eb2f3f13d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:50.523153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.523002    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-workload-socket" pod "65156215-be49-4a1b-a7fd-2b1eb2f3f13d" (UID: "65156215-be49-4a1b-a7fd-2b1eb2f3f13d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:50.523760 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.523732    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istiod-ca-cert" pod "65156215-be49-4a1b-a7fd-2b1eb2f3f13d" (UID: "65156215-be49-4a1b-a7fd-2b1eb2f3f13d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:02:50.524655 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.524631    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-credential-socket" pod "65156215-be49-4a1b-a7fd-2b1eb2f3f13d" (UID: "65156215-be49-4a1b-a7fd-2b1eb2f3f13d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:50.533681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.533259    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"]
Nov 02 18:02:50.533681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.533505    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-data" pod "65156215-be49-4a1b-a7fd-2b1eb2f3f13d" (UID: "65156215-be49-4a1b-a7fd-2b1eb2f3f13d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:50.543452 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.543195    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-557feb-openshift-default-dockercfg-lvqfg\""
Nov 02 18:02:50.544048 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.544020    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-token" pod "65156215-be49-4a1b-a7fd-2b1eb2f3f13d" (UID: "65156215-be49-4a1b-a7fd-2b1eb2f3f13d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:02:50.548645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.548620    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-podinfo" pod "65156215-be49-4a1b-a7fd-2b1eb2f3f13d" (UID: "65156215-be49-4a1b-a7fd-2b1eb2f3f13d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:02:50.549288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.549261    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-kube-api-access-gszm5" pod "65156215-be49-4a1b-a7fd-2b1eb2f3f13d" (UID: "65156215-be49-4a1b-a7fd-2b1eb2f3f13d"). InnerVolumeSpecName "kube-api-access-gszm5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:02:50.549883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.549800    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-envoy" pod "65156215-be49-4a1b-a7fd-2b1eb2f3f13d" (UID: "65156215-be49-4a1b-a7fd-2b1eb2f3f13d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:02:50.620338 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.620305    4456 generic.go:356] "Generic (PLEG): container finished" podID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" containerID="6e6b3298b09594bcb799769b170054e6e96452903ad9152cb54b29e190aafdb3" exitCode=0
Nov 02 18:02:50.620865 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.620836    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4" event={"ID":"65156215-be49-4a1b-a7fd-2b1eb2f3f13d","Type":"ContainerDied","Data":"6e6b3298b09594bcb799769b170054e6e96452903ad9152cb54b29e190aafdb3"}
Nov 02 18:02:50.621044 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.621025    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4" event={"ID":"65156215-be49-4a1b-a7fd-2b1eb2f3f13d","Type":"ContainerDied","Data":"a123bfea1371bbad94ee368639c886b357d4eb175cf741dd99912d1580687e91"}
Nov 02 18:02:50.621738 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.620874    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"
Nov 02 18:02:50.621738 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.621064    4456 scope.go:122] "RemoveContainer" containerID="6e6b3298b09594bcb799769b170054e6e96452903ad9152cb54b29e190aafdb3"
Nov 02 18:02:50.622852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.622786    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-podinfo\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.622916    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-shksx\" (UniqueName: \"kubernetes.io/projected/856c2dca-bad2-4f62-a536-1ca0e09f0a12-kube-api-access-shksx\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.622953    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-envoy\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.622993    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-workload-socket\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.623051    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-data\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.623103    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-workload-certs\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.623151    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-token\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.623301    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-credential-socket\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.623365    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istiod-ca-cert\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.623568    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.623587    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.623669    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gszm5\" (UniqueName: \"kubernetes.io/projected/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-kube-api-access-gszm5\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.623685    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.623698    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.623711    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.623725    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.623738    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:02:50.624230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.623751    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/65156215-be49-4a1b-a7fd-2b1eb2f3f13d-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:02:50.648407 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.648388    4456 scope.go:122] "RemoveContainer" containerID="5d9cae3ca58d585af27bfd3ba6620280816bb0024b13b0622caebfcbac09cf0b"
Nov 02 18:02:50.663409 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.663389    4456 scope.go:122] "RemoveContainer" containerID="6e6b3298b09594bcb799769b170054e6e96452903ad9152cb54b29e190aafdb3"
Nov 02 18:02:50.663719 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:02:50.663699    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6e6b3298b09594bcb799769b170054e6e96452903ad9152cb54b29e190aafdb3\": container with ID starting with 6e6b3298b09594bcb799769b170054e6e96452903ad9152cb54b29e190aafdb3 not found: ID does not exist" containerID="6e6b3298b09594bcb799769b170054e6e96452903ad9152cb54b29e190aafdb3"
Nov 02 18:02:50.663829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.663731    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6e6b3298b09594bcb799769b170054e6e96452903ad9152cb54b29e190aafdb3"} err="failed to get container status \"6e6b3298b09594bcb799769b170054e6e96452903ad9152cb54b29e190aafdb3\": rpc error: code = NotFound desc = could not find container \"6e6b3298b09594bcb799769b170054e6e96452903ad9152cb54b29e190aafdb3\": container with ID starting with 6e6b3298b09594bcb799769b170054e6e96452903ad9152cb54b29e190aafdb3 not found: ID does not exist"
Nov 02 18:02:50.663829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.663758    4456 scope.go:122] "RemoveContainer" containerID="5d9cae3ca58d585af27bfd3ba6620280816bb0024b13b0622caebfcbac09cf0b"
Nov 02 18:02:50.664087 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:02:50.664066    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"5d9cae3ca58d585af27bfd3ba6620280816bb0024b13b0622caebfcbac09cf0b\": container with ID starting with 5d9cae3ca58d585af27bfd3ba6620280816bb0024b13b0622caebfcbac09cf0b not found: ID does not exist" containerID="5d9cae3ca58d585af27bfd3ba6620280816bb0024b13b0622caebfcbac09cf0b"
Nov 02 18:02:50.664124 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.664096    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"5d9cae3ca58d585af27bfd3ba6620280816bb0024b13b0622caebfcbac09cf0b"} err="failed to get container status \"5d9cae3ca58d585af27bfd3ba6620280816bb0024b13b0622caebfcbac09cf0b\": rpc error: code = NotFound desc = could not find container \"5d9cae3ca58d585af27bfd3ba6620280816bb0024b13b0622caebfcbac09cf0b\": container with ID starting with 5d9cae3ca58d585af27bfd3ba6620280816bb0024b13b0622caebfcbac09cf0b not found: ID does not exist"
Nov 02 18:02:50.679852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.679803    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"]
Nov 02 18:02:50.687893 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.687865    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b913b2-openshift-default-69c9976c4c-lhpx4"]
Nov 02 18:02:50.726081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.726050    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-token\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.726261 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.726244    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-credential-socket\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.726333 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.726318    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istiod-ca-cert\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.726539 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.726522    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-podinfo\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.726656 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.726635    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-shksx\" (UniqueName: \"kubernetes.io/projected/856c2dca-bad2-4f62-a536-1ca0e09f0a12-kube-api-access-shksx\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.726841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.726660    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-credential-socket\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.726841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.726673    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-envoy\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.726841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.726754    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-workload-socket\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.727015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.726843    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-data\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.727015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.726899    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-workload-certs\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.727960 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.727633    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-data\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.727960 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.727677    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-workload-socket\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.727960 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.727748    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-workload-certs\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.727960 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.727777    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istiod-ca-cert\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.730500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.730470    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-envoy\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.730726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.730710    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-podinfo\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.758207 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.758182    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-token\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.758343 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.758325    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-shksx\" (UniqueName: \"kubernetes.io/projected/856c2dca-bad2-4f62-a536-1ca0e09f0a12-kube-api-access-shksx\") pod \"e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") " pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:50.850734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:50.850656    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:51.206082 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:51.206054    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"]
Nov 02 18:02:51.208461 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:02:51.208427    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod856c2dca_bad2_4f62_a536_1ca0e09f0a12.slice/crio-9b54f67d672baa65cc06b469385db955c33c287dcbfc717a6514b0c7eda79d35 WatchSource:0}: Error finding container 9b54f67d672baa65cc06b469385db955c33c287dcbfc717a6514b0c7eda79d35: Status 404 returned error can't find the container with id 9b54f67d672baa65cc06b469385db955c33c287dcbfc717a6514b0c7eda79d35
Nov 02 18:02:51.212986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:51.212950    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:51.213181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:51.213039    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:51.213181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:51.213088    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:02:51.622976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:51.622893    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" path="/var/lib/kubelet/pods/65156215-be49-4a1b-a7fd-2b1eb2f3f13d/volumes"
Nov 02 18:02:51.635373 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:51.635338    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" event={"ID":"856c2dca-bad2-4f62-a536-1ca0e09f0a12","Type":"ContainerStarted","Data":"51e90f82024911642a387ee990edbbcce91cbeeb7879f1ae0c57ac3433a8cfc7"}
Nov 02 18:02:51.635558 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:51.635408    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" event={"ID":"856c2dca-bad2-4f62-a536-1ca0e09f0a12","Type":"ContainerStarted","Data":"9b54f67d672baa65cc06b469385db955c33c287dcbfc717a6514b0c7eda79d35"}
Nov 02 18:02:51.697505 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:51.697444    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" podStartSLOduration=5.697426947 podStartE2EDuration="5.697426947s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:46 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:51.69415165 +0000 UTC m=+1988.185214825" watchObservedRunningTime="2026-11-02 18:02:51.697426947 +0000 UTC m=+1988.188490108"
Nov 02 18:02:51.853841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:51.850963    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:51.941662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:51.941624    4456 scope.go:122] "RemoveContainer" containerID="384a689b9c5f882442305b364f7f0d010f84e512893f729c400ab5d63b7eb392"
Nov 02 18:02:51.962521 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:51.962502    4456 scope.go:122] "RemoveContainer" containerID="992d2a3e54d4db746448b02f06786266b9c02e56729836a4f100dc885f70cf7f"
Nov 02 18:02:52.015749 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:52.015416    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:52.857705 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:52.857677    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:53.651168 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:53.651135    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:53.652171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:53.652147    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:02:54.009677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:54.009627    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:54.185514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:54.185446    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:54.371530 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:54.371445    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:55.977127 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:55.977087    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:56.121235 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:56.121193    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"]
Nov 02 18:02:56.122886 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:56.122860    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="65156215-be49-4a1b-a7fd-2b1eb2f3f13d" containerName="istio-proxy"
Nov 02 18:02:56.129459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:56.129435    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"
Nov 02 18:02:56.146218 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:56.146188    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"]
Nov 02 18:02:56.153621 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:56.153594    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8srp8\" (UniqueName: \"kubernetes.io/projected/49238ac8-57fa-412a-88bd-681364a0b455-kube-api-access-8srp8\") pod \"payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2\" (UID: \"49238ac8-57fa-412a-88bd-681364a0b455\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"
Nov 02 18:02:56.153826 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:56.153792    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49238ac8-57fa-412a-88bd-681364a0b455-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2\" (UID: \"49238ac8-57fa-412a-88bd-681364a0b455\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"
Nov 02 18:02:56.163490 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:56.163465    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-557feb-dockercfg-nfpss\""
Nov 02 18:02:56.164513 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:56.164495    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-557feb\""
Nov 02 18:02:56.212864 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:56.190149    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:56.315928 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:56.309388    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8srp8\" (UniqueName: \"kubernetes.io/projected/49238ac8-57fa-412a-88bd-681364a0b455-kube-api-access-8srp8\") pod \"payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2\" (UID: \"49238ac8-57fa-412a-88bd-681364a0b455\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"
Nov 02 18:02:56.315928 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:56.309562    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49238ac8-57fa-412a-88bd-681364a0b455-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2\" (UID: \"49238ac8-57fa-412a-88bd-681364a0b455\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"
Nov 02 18:02:56.421849 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:56.412862    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49238ac8-57fa-412a-88bd-681364a0b455-plugins-config-volume\") pod \"payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2\" (UID: \"49238ac8-57fa-412a-88bd-681364a0b455\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"
Nov 02 18:02:56.758093 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:56.757788    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8srp8\" (UniqueName: \"kubernetes.io/projected/49238ac8-57fa-412a-88bd-681364a0b455-kube-api-access-8srp8\") pod \"payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2\" (UID: \"49238ac8-57fa-412a-88bd-681364a0b455\") " pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"
Nov 02 18:02:57.045907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:57.045784    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"
Nov 02 18:02:57.386703 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:02:57.386665    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod49238ac8_57fa_412a_88bd_681364a0b455.slice/crio-6238a8826930486e5da032e1da913157c2129d65eff9738f90f2276137fb9f80 WatchSource:0}: Error finding container 6238a8826930486e5da032e1da913157c2129d65eff9738f90f2276137fb9f80: Status 404 returned error can't find the container with id 6238a8826930486e5da032e1da913157c2129d65eff9738f90f2276137fb9f80
Nov 02 18:02:57.388721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:57.388677    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"]
Nov 02 18:02:57.776660 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:57.776620    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2" event={"ID":"49238ac8-57fa-412a-88bd-681364a0b455","Type":"ContainerStarted","Data":"1078c01c652fa0ea5d83259df4fe78a4cf54da34d16748e6c4e9933597f21ac0"}
Nov 02 18:02:57.778044 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:57.776679    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2" event={"ID":"49238ac8-57fa-412a-88bd-681364a0b455","Type":"ContainerStarted","Data":"6238a8826930486e5da032e1da913157c2129d65eff9738f90f2276137fb9f80"}
Nov 02 18:02:57.778044 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:57.776709    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"
Nov 02 18:02:57.858837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:57.851516    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2" podStartSLOduration=1.851491793 podStartE2EDuration="1.851491793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:02:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:02:57.819507097 +0000 UTC m=+1994.310570283" watchObservedRunningTime="2026-11-02 18:02:57.851491793 +0000 UTC m=+1994.342554954"
Nov 02 18:02:57.923247 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:57.923215    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:02:59.760279 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:02:59.760249    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:01.584488 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:01.578970    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:03.389161 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:03.389108    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:05.166472 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:05.166411    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:06.967588 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:06.967543    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:08.703508 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:08.703444    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:08.790167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:08.790137    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"
Nov 02 18:03:11.053995 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:11.053936    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:11.069270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:11.069165    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:11.167978 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:11.167925    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:11.363566 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:11.358949    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:11.449860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:11.449480    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:11.632014 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:11.631913    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:11.635005 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:11.633690    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:11.648359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:11.645166    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:11.705842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:11.692506    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:12.170995 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:12.169086    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:12.170995 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:12.169745    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 18:03:12.360996 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:12.360944    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:12.361976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:12.361765    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 18:03:12.645839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:12.593115    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 18:03:12.720065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:12.699157    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 18:03:12.720065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:12.699572    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:03:12.747845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:12.723955    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:12.747845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:12.724314    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 18:03:12.747845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:12.724632    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 18:03:12.747845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:12.724689    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 18:03:12.761262 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:12.761213    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:12.777096 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:12.763324    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 18:03:13.039842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:13.033529    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:13.156853 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:13.126343    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded"
Nov 02 18:03:13.227854 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:13.226922    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:13.311898 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:13.282185    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 18:03:13.497902 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:13.473115    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:13.533007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:13.532936    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:03:13.533007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:13.533006    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:13.729674 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:13.729627    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 18:03:13.862219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:13.843838    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 18:03:13.862219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:13.853954    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:13.878410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:13.878354    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 18:03:13.878785 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:13.878738    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:14.012004 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:13.899681    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:03:14.037841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:14.033936    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 18:03:14.170250 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:14.170212    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:03:14.239510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:14.239470    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 18:03:14.264288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:14.264207    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 18:03:14.264604 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:14.264581    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 18:03:14.288963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:14.288928    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:14.302676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:14.295529    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 18:03:14.315174 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:14.315146    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 18:03:14.322843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:14.320850    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:03:14.333134 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:14.333111    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 18:03:14.345117 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:14.345020    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 18:03:14.345277 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:14.345219    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 18:03:14.347616 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:14.347599    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 18:03:14.352225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:14.352209    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 18:03:15.805830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:15.805692    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:16.220750 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:03:16.220706    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:03:17.694314 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:17.694260    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:03:19.702468 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:03:19.702421    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:37.924518    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.15:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:37.731773    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:37.924852    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:37.925596    4456 patch_prober.go:29] interesting pod/metrics-server-7d987f75-ntp98 container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.16:10250/livez\": context deadline exceeded" start-of-body=
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:37.886321    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:37.959457    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" podUID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.45:50052: i/o timeout"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:37.959941    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" podUID="44b66f41-99ae-446b-a50a-2ec0b71563a4" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.189:50052: i/o timeout"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:37.960162    4456 trace.go:236] Trace[1081423391]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" (02-Nov-2026 18:04:36.688) (total time: 1271ms):
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: Trace[1081423391]: [1.271755293s] [1.271755293s] END
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:37.960275    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.076304    4456 trace.go:236] Trace[920041176]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" (02-Nov-2026 18:04:36.687) (total time: 1389ms):
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: Trace[920041176]: [1.389217358s] [1.389217358s] END
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.479153    4456 trace.go:236] Trace[723405832]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" (02-Nov-2026 18:04:36.638) (total time: 1840ms):
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: Trace[723405832]: [1.840223917s] [1.840223917s] END
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.489765    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.565264    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.170:50052: i/o timeout"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.565643    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.565691    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.565967    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.173:50052: i/o timeout"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.566556    4456 trace.go:236] Trace[320056491]: "Calculate volume metrics of istio-podinfo for pod openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" (02-Nov-2026 18:04:36.690) (total time: 1270ms):
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: Trace[320056491]: [1.270082271s] [1.270082271s] END
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.566889    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.567061    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.566864    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.570921    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.579031    4456 trace.go:236] Trace[1586279973]: "Calculate volume metrics of iptables-alerter-script for pod openshift-network-operator/iptables-alerter-dlp6w" (02-Nov-2026 18:04:36.648) (total time: 1930ms):
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: Trace[1586279973]: [1.930131458s] [1.930131458s] END
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.580244    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.580328    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.583499    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.583536    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.584414    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.586132    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.587648    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-7d987f75-ntp98" podUID="18d2ae0a-bf65-4aee-8730-093e00b54a24" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.16:10250/livez\": context deadline exceeded"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.590221    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.600998    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" podUID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.190:50052\" within 1s: context deadline exceeded"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.601253    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.601290    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.601447    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.601467    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.601678    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.603381    4456 trace.go:236] Trace[1971200411]: "Calculate volume metrics of kube-api-access-kjmhh for pod openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2" (02-Nov-2026 18:04:34.803) (total time: 3800ms):
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: Trace[1971200411]: [3.800318407s] [3.800318407s] END
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.603420    4456 trace.go:236] Trace[1527148364]: "Calculate volume metrics of model-cache for pod e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" (02-Nov-2026 18:04:34.791) (total time: 3811ms):
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: Trace[1527148364]: [3.811472528s] [3.811472528s] END
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.606644    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.43:50052: i/o timeout"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.607332    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.43:50052\" within 1s: context deadline exceeded"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.607652    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.187:50052\" within 1s: context deadline exceeded"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.608493    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.170:50052\" within 1s: context deadline exceeded"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:04:38.609304    4456 kubelet.go:2545] "Skipping pod synchronization" err="container runtime is down"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:04:38.720432    4456 kubelet.go:2545] "Skipping pod synchronization" err="container runtime is down"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:38.720655    4456 logging.go:55] [core] [Channel #5857 SubChannel #5860]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.43:50052", ServerName: "10.134.0.43:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.43:50052: operation was canceled"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:38.720655    4456 logging.go:55] [core] [Channel #5856 SubChannel #5858]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.170:50052", ServerName: "10.134.0.170:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.170:50052: operation was canceled"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:38.720677    4456 logging.go:55] [core] [Channel #5859 SubChannel #5861]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.187:50052", ServerName: "10.134.0.187:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.187:50052: operation was canceled"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:38.720716    4456 logging.go:55] [core] [Channel #5829 SubChannel #5841]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.190:50052", ServerName: "10.134.0.190:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:44346->10.134.0.190:50052: use of closed network connection"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.721693    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.740072    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.740484    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.836672    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m18.008922608s: [/var/lib/containers/storage/overlay/d15dc5bffc34599d4e17b91bff8ac1423d9077ae71856a7a2755b92a3bd6dcfc/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-5knvx_8ef79a10-292b-42d9-aab0-53bee3e43ed2/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.840036    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.176:50052\" within 1s: context deadline exceeded"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.840600    4456 trace.go:236] Trace[456658783]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" (02-Nov-2026 18:04:34.809) (total time: 4031ms):
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: Trace[456658783]: [4.031067654s] [4.031067654s] END
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:38.842981    4456 logging.go:55] [core] [Channel #5882 SubChannel #5883]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.176:50052", ServerName: "10.134.0.176:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.176:50052: operation was canceled"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.843642    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.200:50052: i/o timeout"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.856070    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.210301219s: [/var/lib/containers/storage/overlay/8da893366900c89f1a49f0a89cff3a847547158c42b584792c9ed31f41e77cc2/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.856100    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.856140    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.721400    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.913748042s: [/var/lib/containers/storage/overlay/f9027342e93f93cf2207e54c54a505f7bd2ec4fc18d0fb5e35c550a0d937439e/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.733260    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.218167794s: [/var/lib/containers/storage/overlay/12a0c2eeda822e1ff661410a43076cdd6aedef1ccef4240d7456685f94696af9/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:37.942729    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.096643084s: [/var/lib/containers/storage/overlay/f41fd51d26fbd21de386cefe58613bae1900970a633ac15435e1675fb963123a/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.751389    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.971373713s: [/var/lib/containers/storage/overlay/47c6f9fa4e81d5b6bb6cb4211eff0df7ee33a694a7d6141f384651326b031a13/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.564693    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 24.055822742s: [/var/lib/containers/storage/overlay/0234c330ad731526e0f7c2e86fd1692f4cb5053f776e389d790b7710b7d2e028/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.616417    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.644431299s: [/var/lib/containers/storage/overlay/eea65d58e8e5145ae3222bd58cd763318191e7e1e8a3911be816c6f629e9177c/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.857456    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.17353831s: [/var/lib/containers/storage/overlay/bd48af5b0a05e1435415768044430abaf131384d551707641cd919d54da040a4/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.857510    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.173587758s: [/var/lib/containers/storage/overlay/84e2978c7bf6f8a82e9cb045dcd37b94011845204a16c93bb3912ff1c3d5582e/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.857551    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.173628423s: [/var/lib/containers/storage/overlay/37d56634756aeeeca9171a043730fdb311d7d6352312d621fb8e69abd123ca4d/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.857590    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.173666434s: [/var/lib/containers/storage/overlay/9f3bb7fa13dcbf6c99ec343710a35437e7f61d66d1bdac63538a05c396354291/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.857632    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.173706048s: [/var/lib/containers/storage/overlay/8a61e7e207706ded21f782a468d9622ae8cfb244d32ad9c9d3f18363aca5ad52/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.857671    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.173744371s: [/var/lib/containers/storage/overlay/229a9ce45461e83bcf894d5fdec3589a18c0dde28fd30e6c2f7267725d721843/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.857714    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.17378611s: [/var/lib/containers/storage/overlay/ced9b4ee10b95de9e21ed7fe4450d4c9fcc97bb9e1d3ebda22fd9d469e5f4292/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.857757    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.173827738s: [/var/lib/containers/storage/overlay/0a1b5af1d2fbdef3ff1b7294208f7f4115ccab6c1c6f56a4dbd94ef547fdbb15/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.857800    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.173869024s: [/var/lib/containers/storage/overlay/46c87d9588b5c4153befd46bcce313bb8d9eed33c45f7c4b76fbab7d7567da57/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.857883    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.173943511s: [/var/lib/containers/storage/overlay/f1bf059e369a60e8c8c66e453cf0724779ec1592faffe0ceeaccedee71095804/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.857937    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.173994038s: [/var/lib/containers/storage/overlay/959687ac2acb5f81ac7bb8a093e2668af6b0018b8cabcc37179678695dd639a3/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.857979    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.174036248s: [/var/lib/containers/storage/overlay/1c9f6f61c3280851db3a1425f487b201d7a593a4e27a560538129810b3169adc/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.858021    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.174076703s: [/var/lib/containers/storage/overlay/b428ce3434de02ee9be0b02f069e7c0e10cfc97ead5bb22bcc622bbc6836f7d3/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.858064    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.174120013s: [/var/lib/containers/storage/overlay/e64870204381c7a7f2555e3b5fe92951842ced33934d9f090c927aa4bb74212e/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.858104    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.174158341s: [/var/lib/containers/storage/overlay/a1ee19348961dae79c087e8ac3bf9e59ed7a33f37091eafee1f692eee68bc0a0/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.858148    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.174213925s: [/var/lib/containers/storage/overlay/b47b9dd6c514448c5e5a1ae04c109b9960c3981afb61d47419b7ec217051ba27/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.858172    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.858200    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.174251605s: [/var/lib/containers/storage/overlay/9a65488a0d46fd5c20f43cfe794eb9f7883b34fba7149707cfc381b54e62c6dd/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.858244    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.174294444s: [/var/lib/containers/storage/overlay/2b04fc8b211dff465dfc3422cdf939d7911c23386635e5015c53667136caa0f0/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.858297    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.17434554s: [/var/lib/containers/storage/overlay/7e3710c7c30531a95b95d756958e640520b26b432243904fffd62b97f6e3862f/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.859969    4456 trace.go:236] Trace[2138408887]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" (02-Nov-2026 18:04:34.792) (total time: 4067ms):
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: Trace[2138408887]: [4.06737608s] [4.06737608s] END
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.860030    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:38.861057    4456 logging.go:55] [core] [Channel #5886 SubChannel #5887]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.171:50052", ServerName: "10.134.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.171:50052: operation was canceled"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.862757    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.177221266s: [/var/lib/containers/storage/overlay/2848def6302080170b0488e9cb4ed92ffb69bd4f38ed3e4eba1e06c02f7c02fa/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.866191    4456 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-pgmzg container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.866261    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:04:48.010157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.866429    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": context deadline exceeded" start-of-body=
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.866506    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": context deadline exceeded"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.866673    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.867888    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.868967    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.134:50052: i/o timeout"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.869222    4456 trace.go:236] Trace[8546344]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" (02-Nov-2026 18:04:35.450) (total time: 3418ms):
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: Trace[8546344]: [3.418620751s] [3.418620751s] END
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.872714    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" podUID="e4c1132f-05b5-4664-84a2-ece57727b12a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.122:50052: i/o timeout"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.874177    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" podUID="d837dde2-3939-4400-8ea7-fee58c162983" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.116:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.875585    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" podUID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.183:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:38.876335    4456 logging.go:55] [core] [Channel #5900 SubChannel #5902]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.183:50052", ServerName: "10.134.0.183:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.183:50052: operation was canceled"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:38.876502    4456 logging.go:55] [core] [Channel #5896 SubChannel #5897]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.116:50052", ServerName: "10.134.0.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.116:50052: operation was canceled"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.878546    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" podUID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.182:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:38.879072    4456 logging.go:55] [core] [Channel #5832 SubChannel #5906]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.182:50052", ServerName: "10.134.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.182:50052: operation was canceled"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.881505    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" podUID="e4c1132f-05b5-4664-84a2-ece57727b12a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.122:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.881546    4456 trace.go:236] Trace[340563939]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" (02-Nov-2026 18:04:35.465) (total time: 3416ms):
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: Trace[340563939]: [3.41610088s] [3.41610088s] END
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.882174    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.134:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:38.884328    4456 logging.go:55] [core] [Channel #5830 SubChannel #5911]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.122:50052", ServerName: "10.134.0.122:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.122:50052: operation was canceled"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.886121    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.886158    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.887939    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.887977    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.890281    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.010003086s: [/var/lib/containers/storage/overlay/578d0af976732c1c9e43c84eae06aa0aa97b984f5931bc98ceceb74deb7f87b2/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.893131    4456 ???:1] "http: TLS handshake error from 10.0.136.153:34180: EOF"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.893966    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.007774619s: [/var/lib/containers/storage/overlay/ec89134dec345974daece69f9b8dd6137e843685bffddf149e15d295f298817e/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.894166    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.007974159s: [/var/lib/containers/storage/overlay/b85fed92aa7942c2d1b0a3ebd90904062437ad83478c28e6d785342de059e340/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.894210    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.008018331s: [/var/lib/containers/storage/overlay/8099f23161d5eec2e26bd9fd0d29e0723c872c3a2553491cef9562905ddcb497/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.894249    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.00805571s: [/var/lib/containers/storage/overlay/25756dda2621604e75c6eea74f5fb75349668aa5a63ad77893fd3405a7c6f263/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.894298    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.00809658s: [/var/lib/containers/storage/overlay/c54941b7bb0f708eefe849545e48082649e74c106518b009fddf3e0544c6bc0f/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.894338    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.008134228s: [/var/lib/containers/storage/overlay/f43459bfc40568911db1c28d46def37ad81ebdd789dcbc7aef804efa5574b108/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.894376    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.008135649s: [/var/lib/containers/storage/overlay/7530ca161075d72465e087d6cfea7fe9079bd67c737753ce9107f7d5bf04700b/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.894636    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.895185    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.895259    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.897732    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.256484542s: [/var/lib/containers/storage/overlay/0d74f9f4fbbf600087eeb718bd1494b6cee4f94be2e06fed79e7cc0e32173cf4/diff /var/log/pods/openshift-dns_dns-default-zmrdt_f59eba5d-a500-462e-9ecd-4ef9e553f873/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.911191    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.911230    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.912017    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.912048    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.912966    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.913007    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.915991    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.134.0.28:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.916021    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.917037    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.917064    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.918176    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.918212    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:04:38.920607    4456 kubelet.go:2545] "Skipping pod synchronization" err="container runtime is down"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.921142    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.034523864s: [/var/lib/containers/storage/overlay/4e361aab9892c59ae1c857a4c3f26675ab711d42dbb2eea22c0038a24983687a/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.921215    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.03459832s: [/var/lib/containers/storage/overlay/88bc22348df8e280411872f066dbf14feb3968b22f192d53027960ffae02080e/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.921254    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.034638281s: [/var/lib/containers/storage/overlay/a7f26589ec75d499102c1e32470fe7d9c2e016680764619bc4f468b50424ee5f/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.921485    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.459459803s: [/var/lib/containers/storage/overlay/1f335257c28f61349dc6c4495b958b214a44a29fd9b6b29ce082d223b3fa08f9/diff /var/log/pods/openshift-insights_insights-runtime-extractor-6bfcf_da9f1b66-0e1b-4c7e-a947-a897c00ccc19/extractor/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.921528    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.034898502s: [/var/lib/containers/storage/overlay/7cd1c406d79080a5de52020433aba1bd1e3439731a6008c91a44ba0d734e0168/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.921583    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.034951289s: [/var/lib/containers/storage/overlay/7a274702c3ffdc2535cd43c3814dd50ea0416d2fecf6b613833a80a6cd3be072/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.921621    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.034988922s: [/var/lib/containers/storage/overlay/889a1e50cd346c2ae6c3699e22fc6333114a9fef99ca78329cab0059d0af21f8/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.921659    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.035027357s: [/var/lib/containers/storage/overlay/12034bda1ff91295c47a5affae545ed5da549ac45cf44e53b8bd75905db1b189/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.921697    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.035064362s: [/var/lib/containers/storage/overlay/e4f11f93cbfd2fc79389aec87c75cc993913f4c053e02fc2bef087f73afd5d7d/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.921722    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.202157341s: [/var/lib/containers/storage/overlay/f1530aa35ef7889a1d1dc0c9b37e6279c014c286fd6a5858a1a15daaefad8625/diff /var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-q2mbj_cae854c8-88f7-4201-b5d7-4e7c59fdf833/prometheus-operator-admission-webhook/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.922590    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.922629    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:38.923100    4456 logging.go:55] [core] [Channel #5831 SubChannel #5912]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.134:50052", ServerName: "10.134.0.134:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:51418->10.134.0.134:50052: use of closed network connection"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.926159    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.926213    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.928141    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.928399    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded" start-of-body=
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.928437    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.931194    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.044540196s: [/var/lib/containers/storage/overlay/a59d3ee737f274b70a661f619f4db78532081f4b90d92849748b1bc8db60f573/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.931241    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.044591024s: [/var/lib/containers/storage/overlay/4e67e8cd7fb897bc5116e504bba9a29a8ed30c4f9f1fac62d47ec5892d590847/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.931287    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.044627229s: [/var/lib/containers/storage/overlay/a18f6543a5970d2a5877e9cb019fa68742596c240618559891255c531214ebc1/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.936550    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.049847911s: [/var/lib/containers/storage/overlay/510d5b5021012ada853fd2e3b25638149887bac7a79939d0bf56c0262cc66081/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.936607    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.04994094s: [/var/lib/containers/storage/overlay/c7363f8222650c51f900a246744d12fb88176e629d384d106cefbae7c47dd7cd/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.936643    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.049977103s: [/var/lib/containers/storage/overlay/2a7f5aa5ba39a921bd515553a1089c5e8bb4a586c47eda778d4017ed30ecc198/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.941294    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.941355    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.943788    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.057105701s: [/var/lib/containers/storage/overlay/a5abae015c3465904cc5c7174a5b19880c528412ca15a33a710b819431e1c2f7/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.962229    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.075538953s: [/var/lib/containers/storage/overlay/1a394e1eaa16f81c47d4d9f50cab7c43ca075ce2f22fa2906b7916363672c95c/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.980041    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.092846429s: [/var/lib/containers/storage/overlay/0024ef56118e59543f04eba902323a6c84cea8671f97bb98441651fafd3f1fb9/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.980196    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.092657536s: [/var/lib/containers/storage/overlay/18864169905dab44c7b7c46db110c9b2e1d7a5b103538ab226d089b5c195ecec/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.988187    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.100527827s: [/var/lib/containers/storage/overlay/c155031b458209f5c3cd2b2c6dbb1583f454b4f2ea510626f12b6fd5ab84995e/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.988678    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.100978508s: [/var/lib/containers/storage/overlay/2e0adf259b4f46a31705cecfb5e7924b04dcec1097b7bec6afcbc60c92bfa8a4/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.988715    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.101014936s: [/var/lib/containers/storage/overlay/cb2f4d21c7e74b313922bb09d6b5f3805778f08b0046d9b0e26544665af451da/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.988882    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.101178762s: [/var/lib/containers/storage/overlay/fab27cad89ad4b3e8d0b5c4ac0b7b3a7912aa4fa9886a06d4abd5720cb2eb720/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.988916    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.101213385s: [/var/lib/containers/storage/overlay/76e05be6c6727d8ded946e030da1c274a89ca48d16a9caf69ad1842aaef2e487/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.988942    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.101228708s: [/var/lib/containers/storage/overlay/655a7e468823c9c970918a567d0b0ed61e5e4f40137739e6fa8ac19b456819a6/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.989123    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.101407238s: [/var/lib/containers/storage/overlay/491769e17d91cf9015f229933a02ba515455f7a916be79b9866f14aaeb3a54d7/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.989151    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.101427296s: [/var/lib/containers/storage/overlay/02d4c961dc5c71ea22ce464a653f70d8ea7afa2333251e6e0f32faecb297ca22/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.989185    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.101459652s: [/var/lib/containers/storage/overlay/5eda4bcefdda7ae7749489bd6e8252ea64d62d114e01e22912d64207986209b0/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.989213    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.101486715s: [/var/lib/containers/storage/overlay/1a8ec8fdfc84ff22ef5720fbfd7b513681a22a4a90a63c80b66dec40cf96c5a9/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.989238    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.101511011s: [/var/lib/containers/storage/overlay/b08034c65637061c7d06856c8763d8af38fa0083db3da8a46ee71187bc8aa3f5/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.989271    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.101542971s: [/var/lib/containers/storage/overlay/6db68c524c0be1eea698df87a8b9a3de739d2a7cafcb2c4289aec09a3b1cfe4f/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.989457    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.101678432s: [/var/lib/containers/storage/overlay/48b0c974b21e6e5c0615a44fe7b2a01a2a0049cec1284df8211410724ae86bc8/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.989492    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.101711185s: [/var/lib/containers/storage/overlay/76420521e8040790a74114ea6b6329940e29b9c3789f480a932d32ff1a407957/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.989521    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.101740598s: [/var/lib/containers/storage/overlay/7f6de02a430f8b7e33a65f8a46c453dc84b8b909c0cab1337c53ba4b1fe6ebc3/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:39.027362    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.139431833s: [/var/lib/containers/storage/overlay/e7f6af957917b0de1eac2c0f835eac9f4d66364bc548a8587261fc6bec5607da/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:38.604287    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.69341392s: [/var/lib/containers/storage/overlay/6f5c404cd07ccb0dab4043262d048165e61ce9e08c344113654e44d09bdeb5d3/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:06.525906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:39.035194    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1.147214757s: [/var/lib/containers/storage/overlay/c7562bd346d831342663c585fc20338a2b2e898c53786e7fb087a0324f4f7e13/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:39.037896    4456 trace.go:236] Trace[166276121]: "Calculate volume metrics of env-overrides for pod openshift-ovn-kubernetes/ovnkube-node-44l94" (02-Nov-2026 18:04:35.461) (total time: 3575ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[166276121]: [3.575957103s] [3.575957103s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.700092    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.700141    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.700346    4456 trace.go:236] Trace[1396002352]: "Calculate volume metrics of registry-certificates for pod openshift-image-registry/image-registry-59767d8795-7vwkt" (02-Nov-2026 18:04:38.907) (total time: 17792ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[1396002352]: [17.792640504s] [17.792640504s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.700899    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.701782    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.719049    4456 trace.go:236] Trace[346712888]: "Calculate volume metrics of serving-certs-ca-bundle for pod openshift-monitoring/telemeter-client-79c994cf5c-ckj5h" (02-Nov-2026 18:04:38.910) (total time: 17809ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[346712888]: [17.809024876s] [17.809024876s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:04:56.720850    4456 webhook.go:269] Failed to make webhook authorizer request: webhook call failed: context canceled
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:04:56.720869    4456 server.go:358] "Authorization error" err="webhook call failed: context canceled" user="system:kube-apiserver" verb="get" resource="nodes" subresource="proxy"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.742081    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.773654    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.774886    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.775127    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.775330    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.776806    4456 patch_prober.go:29] interesting pod/metrics-server-7d987f75-ntp98 container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: [+]log ok
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: [+]poststarthook/max-in-flight-filter ok
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: [+]poststarthook/storage-object-count-tracker-hook ok
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: [-]metric-collection-timely failed: reason withheld
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: [+]metadata-informer-sync ok
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: livez check failed
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.776881    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-7d987f75-ntp98" podUID="18d2ae0a-bf65-4aee-8730-093e00b54a24" containerName="metrics-server" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.777106    4456 patch_prober.go:29] interesting pod/metrics-server-7d987f75-ntp98 container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: [+]log ok
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: [+]poststarthook/max-in-flight-filter ok
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: [+]poststarthook/storage-object-count-tracker-hook ok
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: [-]metric-collection-timely failed: reason withheld
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: [+]metadata-informer-sync ok
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: livez check failed
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.777128    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-7d987f75-ntp98" podUID="18d2ae0a-bf65-4aee-8730-093e00b54a24" containerName="metrics-server" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.785243    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.859984688s: [/var/lib/containers/storage/overlay/9dcf2fbc8f2a12b867ec625655279863edbfabf7bb03acffd8482f4d3a0eae26/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.786185    4456 trace.go:236] Trace[542004702]: "Calculate volume metrics of openshift-service-ca for pod odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" (02-Nov-2026 18:04:38.905) (total time: 17880ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[542004702]: [17.880251944s] [17.880251944s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.787075    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.861818796s: [/var/lib/containers/storage/overlay/1b20b5f4508512ef1b3778f562c5f97da26da6bf4d0856c4ef9a07f44819b3f0/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.787113    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.861857944s: [/var/lib/containers/storage/overlay/4756e8c484bb99f308a787d9771c513734c834c31e16597655b029fb5dc0d243/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.787156    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.861899585s: [/var/lib/containers/storage/overlay/7f098c561ddc879a9311701be103276e1d03f82ab3efa34f2b873a3e424f0d21/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.787191    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.861933615s: [/var/lib/containers/storage/overlay/da83a0515dd606978929d985eda34e4783473196836c68c43bf562fe9466bd7b/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.787607    4456 trace.go:236] Trace[465277278]: "Calculate volume metrics of serviceca for pod openshift-image-registry/node-ca-lhwgm" (02-Nov-2026 18:04:38.911) (total time: 17875ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[465277278]: [17.875716135s] [17.875716135s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.787982    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.862713302s: [/var/lib/containers/storage/overlay/4aff5b253dee34e4a614e7a9c0214a5c12d6acd592dd21d04cb698f2042a937c/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.788958    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.863647983s: [/var/lib/containers/storage/overlay/6304d0fceeb709e85ca572090857d08706f669cb1c3222985b10ace2393d51cf/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.789990    4456 trace.go:236] Trace[1952016581]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" (02-Nov-2026 18:04:38.921) (total time: 17868ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[1952016581]: [17.868676612s] [17.868676612s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.790883    4456 trace.go:236] Trace[2045597538]: "Calculate volume metrics of prometheus-k8s-rulefiles-1 for pod openshift-monitoring/prometheus-k8s-0" (02-Nov-2026 18:04:38.921) (total time: 17869ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[2045597538]: [17.869580417s] [17.869580417s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.791608    4456 trace.go:236] Trace[94368473]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" (02-Nov-2026 18:04:38.911) (total time: 17879ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[94368473]: [17.879827564s] [17.879827564s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.793876    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 18.868540776s: [/var/lib/containers/storage/overlay/64b4d87597c9acea5f8f6d9f7c7e5728a5c07b72e1d2ec2c99f1b9a6edfbacb0/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.795897    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.15:9091/-/healthy\": context deadline exceeded" start-of-body=
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.795935    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/healthy\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.796189    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.798459    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.798904    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.799040    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.799198    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.800462    4456 patch_prober.go:29] interesting pod/dns-default-zmrdt container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.801148    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-zmrdt" podUID="f59eba5d-a500-462e-9ecd-4ef9e553f873" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.801691    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.806084    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.806134    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.806317    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.806392    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded" start-of-body=
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.806416    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.806682    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.835918    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.835938    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.835921    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-hr8vz" podUID="e73be583-2a26-4379-8169-0a6b8784a11b" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.134.0.26:9403/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.837963    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.844000    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" podUID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.844896    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.846010    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.846018    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.846021    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.848916    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.920861    4456 trace.go:236] Trace[1457580323]: "Calculate volume metrics of istio-csr-ca-configmap for pod openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" (02-Nov-2026 18:04:38.909) (total time: 18011ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[1457580323]: [18.011342108s] [18.011342108s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.971721    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.971777    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.972751    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.029530723s: [/var/lib/containers/storage/overlay/571669d5d66ff820dd233ab868967067d48bbd1c815de34fbc4a6936fddd205b/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.973554    4456 trace.go:236] Trace[1627210403]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" (02-Nov-2026 18:04:38.919) (total time: 18054ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[1627210403]: [18.054421952s] [18.054421952s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.977784    4456 trace.go:236] Trace[238912552]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" (02-Nov-2026 18:04:38.918) (total time: 18059ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[238912552]: [18.059256826s] [18.059256826s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.983250    4456 trace.go:236] Trace[286917827]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" (02-Nov-2026 18:04:38.919) (total time: 18063ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[286917827]: [18.06330083s] [18.06330083s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.986856    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.986916    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.987725    4456 trace.go:236] Trace[807516585]: "Calculate volume metrics of manager-config for pod openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" (02-Nov-2026 18:04:38.922) (total time: 18064ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[807516585]: [18.064830949s] [18.064830949s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.988045    4456 trace.go:236] Trace[316855734]: "Calculate volume metrics of openshift-service-ca for pod odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" (02-Nov-2026 18:04:38.582) (total time: 18405ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[316855734]: [18.405552464s] [18.405552464s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.989689    4456 trace.go:236] Trace[1254200205]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" (02-Nov-2026 18:04:38.911) (total time: 18078ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[1254200205]: [18.078550578s] [18.078550578s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.990162    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-mxsjq" podUID="367d8d85-a5e8-462d-b1d7-91a01e62b726" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.990729    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.047431691s: [/var/lib/containers/storage/overlay/fc1c8fde1913f2a15abbede07a2b0aad79a7cfd6834498affa92bcdb8a60d09b/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.990732    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": context deadline exceeded"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.991127    4456 trace.go:236] Trace[1865237845]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" (02-Nov-2026 18:04:38.911) (total time: 18079ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[1865237845]: [18.079827906s] [18.079827906s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.991404    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 45.446900748s: [/var/lib/containers/storage/overlay/45cb59ff7e98070f3f64e18d636011e6dccd20a5381c66bd78b4eb98321141d9/diff /var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6jpq8_a1ea3dac-7073-4564-b247-89ea94a64342/manager/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.991421    4456 trace.go:236] Trace[1135405960]: "Calculate volume metrics of tmp-dir for pod openshift-dns/node-resolver-hhsrb" (02-Nov-2026 18:04:38.920) (total time: 18070ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[1135405960]: [18.070864161s] [18.070864161s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.991449    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.031017962s: [/var/lib/containers/storage/overlay/9a23cb84ed43c601bc64e8f0c1b8d1f50c37c3e6f03c9342bea4efea4552db90/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.991514    4456 trace.go:236] Trace[1504572750]: "Calculate volume metrics of ovnkube-script-lib for pod openshift-ovn-kubernetes/ovnkube-node-44l94" (02-Nov-2026 18:04:39.041) (total time: 17949ms):
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: Trace[1504572750]: [17.949523044s] [17.949523044s] END
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.993934    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.995316    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.034717411s: [/var/lib/containers/storage/overlay/8db47c776480e39a5f679a3dfbd86e84aed657e023ade55f978a8f68f01ea9f1/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.995365    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.034702366s: [/var/lib/containers/storage/overlay/48451e9990851b8e024d8b3972a86ef0a0c9f6439dec07d6b3ef16f8fae2c355/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.995401    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.034655558s: [/var/lib/containers/storage/overlay/08a91d46928182b4e183ee7ace17ce789177b554acc520da9618f6d1c993924e/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.995439    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.034691101s: [/var/lib/containers/storage/overlay/2269cfd2edca6d6508ccccdbd523a9546a86e6f9de9c2f183ed3f2e8f50fb2fa/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.998102    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.037346472s: [/var/lib/containers/storage/overlay/7049a5acdc2384d17d12da111ac7b7e2c0312fd1c17bd071788f8bf2808c768a/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.998378    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.037622342s: [/var/lib/containers/storage/overlay/95d0719995519e96ca4e2ef099d1cede945650fd1a2d8d4c03c7de64145ce3de/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.998647    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.037865761s: [/var/lib/containers/storage/overlay/e63da92cc01fe6e396c33175c254611578c02ed94dc2baa9164216cbb22bf034/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.998712    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.037926562s: [/var/lib/containers/storage/overlay/b193bcf543ea8163b7da56ebffe1a592dd1c275d976dd60b8d19c46f9f897138/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.998746    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.037961998s: [/var/lib/containers/storage/overlay/b4590eecb2fb63bb3d6ae41839b6301350c179ba2a20691909813f55406e0826/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.998773    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.03798707s: [/var/lib/containers/storage/overlay/4ab2f93d76b331d264037ae17bd9910369889be014d74845e96039454c48bbf7/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.998797    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.038006564s: [/var/lib/containers/storage/overlay/4835e8154ff3f70d907d90cedc5ad8524fba4d2106f30253d83cdcf05ba99c46/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.998840    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.038046686s: [/var/lib/containers/storage/overlay/d3bb1d8cdab5766b606d96bd1ebc61c634339df13a703533522356720f1907ac/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.998917    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.03811962s: [/var/lib/containers/storage/overlay/1c9abd01101f122b64b87bfa59945cc9d609da8965b554240c8857b6517c873e/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.998964    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.038165574s: [/var/lib/containers/storage/overlay/071d5a818923690326943183a456d5f42cc7cf06a45bbf46dce225490af0ccde/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.998996    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.038136503s: [/var/lib/containers/storage/overlay/8a34801be51a556958c8d17b00f8e1380149782e0320cbe6b256572a3ef0e18e/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999026    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.038163727s: [/var/lib/containers/storage/overlay/c3f17c66c7e230b3ba16f42aafdd369608de60e44df877a9cb942d9ed4c41247/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999056    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.038192249s: [/var/lib/containers/storage/overlay/1f6300bdb8f1111be5619f5fe0108ffbf0081e9e27bd7de967b906b8b3ad33d1/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999082    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.03821609s: [/var/lib/containers/storage/overlay/5517713a95fc08e0575503ff1195bf3084c018054f1947590ab93829560900bd/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.284076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999107    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.038212156s: [/var/lib/containers/storage/overlay/76d53787e52a6a6617da848277a740036843744ce34ec997ac569eda8823e979/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999278    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.038375885s: [/var/lib/containers/storage/overlay/4a89652208b5a1fd16b1a02605f67a9b20d35811800ea0052f9e3a8ea729ac7c/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999306    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.038398569s: [/var/lib/containers/storage/overlay/39e7e1a3433145ba62e41a80782f5b1091cd38d1872bf5a708709525a34f7ff7/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999343    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.038433514s: [/var/lib/containers/storage/overlay/78f0ff08138ee83d62f74d0f093a65703c9dba5248bba14dc94fce7c62d2eb68/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999369    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.038442159s: [/var/lib/containers/storage/overlay/02ed3aed9d7d1f9932504189ec7d2043ab6ef5260de12ca49e1cf4bcedd555cf/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999385    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m17.152189442s: [/var/lib/containers/storage/overlay/65d81a0f3b76a7c7a9475e4c40dceab09e6a9c9e98a2e2e3c4cc474c726a6216/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45_7e09749e-e648-40a3-a5f1-f0dc35fccf24/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999406    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 58.695536344s: [/var/lib/containers/storage/overlay/7b63acde62291120f32b79155fd4ac7703b60aa331f58870ab39e3003a8008d6/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb_cf85b137-2b1e-4e22-8f6e-744a02ea4e46/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999427    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.417434937s: [/var/lib/containers/storage/overlay/9012fabf6c9d37acb0fd084cebc4c8dd6d693a7cba8e42b02de88e893ac7cada/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999440    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 53.647382077s: [/var/lib/containers/storage/overlay/735f0e10d9df130fc8a5a1c199498247b93c3829feaab94f9cf3d5136c7492bd/diff /var/log/pods/openshift-image-registry_image-registry-59767d8795-7vwkt_0db36d92-4712-4338-958a-ebabf2c137ba/registry/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999448    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.448493293s: [/var/lib/containers/storage/overlay/ff286283e890b2f02d7b35cbc69ffb7a37f1ddbe5461c9aabad15b3d6f880cd6/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/kube-rbac-proxy-thanos/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999463    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 55.852102981s: [/var/lib/containers/storage/overlay/6df1c22d4bb429d567a6e21956e408dae481462cd854f2449182902dce8fee38/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg_96db7b02-b878-435a-869e-1d30e147c61d/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999484    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 40.743221569s: [/var/lib/containers/storage/overlay/7543a86d2f7dc71bebf4306a35f19682692a0213b5009a3a8a2a0250a1d74670/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/thanos-query/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999474    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 37.617576685s: [/var/lib/containers/storage/overlay/815d874b17f362f32c184d09c5fa4f25b24a51f559e41679c618ca076d1f092d/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy-metrics/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999504    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.728460546s: [/var/lib/containers/storage/overlay/7c41892d8c349d1ad9ae2bbe72fa8e68344bac8db842e0614212577491c66d1d/diff /var/log/pods/openshift-monitoring_telemeter-client-79c994cf5c-ckj5h_e065a5c3-95fd-4ac8-a287-8e1d20219201/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999518    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 45.455008821s: [/var/lib/containers/storage/overlay/f4bf622192145887ae6ee7c0202271dd59950c44754a850ccabfcb18cbe5c0a3/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-m89vg_9257d59b-6aaf-423e-b7ed-2e64dbdfa535/kube-rbac-proxy-self/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999527    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.027550968s: [/var/lib/containers/storage/overlay/9437d7121cf0584354b918af2df36d1e6733adeb8e3ad8e653ee51e2eaf2b268/diff /var/log/pods/e2e-models-e2e-worker-w1-120394_premium-w1-120394-kserve-669cf87f88-rmj4s_5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999542    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 22.201399718s: [/var/lib/containers/storage/overlay/da5b82203a9243942a4be5d58ab45616354c2ad53c09072b04c350a996fc6bd7/diff /var/log/pods/openshift-ingress_e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s_5bb882f6-fd8f-46ae-9396-b52498c5d9db/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999552    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.902532142s: [/var/lib/containers/storage/overlay/3c75f5f1fa0b98e778e718536ee383cfcc5ca413d1084810926fa3df6d132250/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5_02a2dbe6-baeb-40c4-be0f-835239be331f/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999556    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.728523201s: [/var/lib/containers/storage/overlay/44d9303590300b2067d19221719afe781447e01e7a3698b47e26576b04692428/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/config-reloader/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999572    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.530302975s: [/var/lib/containers/storage/overlay/436b55b989bc697feca6cac52a8bbeabf4607101e8e2d4cf93eca0342b9c313b/diff /var/log/pods/openshift-monitoring_node-exporter-r45c5_8c89d394-c878-4069-b3a9-cd0abc822fec/node-exporter/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999573    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 33.599411751s: [/var/lib/containers/storage/overlay/6ea0c1ff6b4e328bbe49837fbf2858fecebf87243389fc2d2d8ab2b78934c93e/diff /var/log/pods/openshift-monitoring_telemeter-client-79c994cf5c-ckj5h_e065a5c3-95fd-4ac8-a287-8e1d20219201/telemeter-client/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999586    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 44.645922014s: [/var/lib/containers/storage/overlay/cf8db5461455b84412026ed2fb5a9f7f27755e4db156859b09da808cfbc028e3/diff /var/log/pods/e2e-models-e2e-worker-w5-863b87_distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f_bffc7364-3157-46e6-8fba-616f128b60cf/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999590    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 58.632574456s: [/var/lib/containers/storage/overlay/54eb6dcdb0f244dbd1ab517bf5fd9b7b8ca4a049b4741da30b72d995007c9bd3/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g_44b66f41-99ae-446b-a50a-2ec0b71563a4/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999600    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.534955378s: [/var/lib/containers/storage/overlay/a568d4ed72e2a436fe819cab66f62bd95c9b47563cb98179484da40963eb831c/diff /var/log/pods/e2e-models-e2e-worker-w3-5eedab_premium-w3-5eedab-kserve-786db78999-f8lbq_f563607b-3be4-4194-aa10-81e8bf2cbf6e/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999610    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.473284842s: [/var/lib/containers/storage/overlay/fdf2869abb29b990967a0fcf0bb0ef8a663dfc34c944a5841012b183fb830979/diff /var/log/pods/openshift-ingress_e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4_92ad0220-435f-4ae8-b743-3421ce98e6fb/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999616    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.484529564s: [/var/lib/containers/storage/overlay/77ea683c9e19434f89b9f60b237a210ef425f091d8706ed7f1ae9292bd9873db/diff /var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-76f76b79c-7n462_ec905c34-6dfb-483f-bb3d-7ba412262584/manager/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999631    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m9.730959279s: [/var/lib/containers/storage/overlay/d6bd7fb5658266e0883715f2708154a35f91862981569c388e3cb5f866afeeb3/diff /var/log/pods/openshift-ingress_e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7_f56cfa88-cc79-4d08-9111-b47d940533ba/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999636    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 37.376589665s: [/var/lib/containers/storage/overlay/b0a7b4a3c01468345eb0ee14fde5db232133703cf3597d505ebb0820a687d86c/diff /var/log/pods/openshift-ingress_e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b_a4323971-4e6e-43f6-bc29-49cbe5f12949/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999652    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.31574374s: [/var/lib/containers/storage/overlay/92e435ce4ca5afe43253221b15bf185141e8b9ccff9215e533a29abb68c8d01e/diff /var/log/pods/openshift-network-operator_iptables-alerter-dlp6w_6767bd29-3165-4e67-81b3-e320e8324034/iptables-alerter/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999657    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.315711144s: [/var/lib/containers/storage/overlay/349bb891311b428d7245edc3a45686b0ebfaa1beb5ddb02e1c5186f29d1a0a38/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/nbdb/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999677    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.315371588s: [/var/lib/containers/storage/overlay/49fe87eab95fb5bcdb653a69186242a7bf84794358441a45b2ffc81148fc7f68/diff /var/log/pods/openshift-multus_network-metrics-daemon-tk8ht_dd733778-7002-4a26-a4e8-4525d8f6bee9/network-metrics-daemon/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999682    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 39.362563004s: [/var/lib/containers/storage/overlay/9f31774d55e55572596ad5b05f6824dca197d8d645a3a2091b071095f5fc348f/diff /var/log/pods/openshift-ingress_e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j_2dc48441-ccc0-47fc-abe7-7c327559eba4/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999700    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 42.891108781s: [/var/lib/containers/storage/overlay/837e2e8c2d7c95b6015e40fc5ad8bcc348a2aaccbb5d63de3e44916d84e01aef/diff /var/log/pods/e2e-models-e2e-worker-w1-120394_distinct-2-w1-120394-kserve-766594cbd8-r7nr9_57a419b6-5333-4b2c-9f01-051dc82c8707/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999706    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m4.800848601s: [/var/lib/containers/storage/overlay/9de66e4fccd5433ca81c75c6841035983055f1565dac16f5b02b99f158a7e535/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5_3d398f17-735a-4d4b-b1cc-5b24fd266535/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999723    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.530455828s: [/var/lib/containers/storage/overlay/f62221cc1bb4ee783a4479f772f1b3eaee2ae9132e39581452d31149674b631a/diff /var/log/pods/e2e-models-e2e-worker-w3-5eedab_simulator-w3-5eedab-kserve-795c4575bb-f25x5_0c6cec58-b8d9-4856-9446-811407a747ab/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999734    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m2.533439169s: [/var/lib/containers/storage/overlay/57432df54ab94c322fcc3f8ed7e5d69537ef205267b7e8198fece234918f9d44/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n_85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999747    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 42.684052323s: [/var/lib/containers/storage/overlay/30c6ef23d54157e2314449730cec7e09831a7cd2d10ec7e7cdc43a4ced04dd2a/diff /var/log/pods/openshift-monitoring_node-exporter-r45c5_8c89d394-c878-4069-b3a9-cd0abc822fec/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999759    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m7.746429105s: [/var/lib/containers/storage/overlay/4123e1fe79aad53819488a2e30b9e50ef9b63a0dbecb73b41827d800cfdcd4c0/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m_9051e10f-854f-4cc8-90a7-dda319c49c9a/maas-api/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999768    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.311099674s: [/var/lib/containers/storage/overlay/411d81d088de93b0674ea4f2dd71b25e85f37341baf36fa8546cc1221ec492ad/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999790    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.311118759s: [/var/lib/containers/storage/overlay/4e57f3a2be8d4ab9b3c25af8d27d43230ec513dda8164ae6c781b961a859b92b/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/prometheus/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999780    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 55.122277436s: [/var/lib/containers/storage/overlay/42bab2822c3e5230896f9c86b84693de76f186c67813e61eb7792866accc56b3/diff /var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4_6e44698a-838e-4eea-b956-585673963ed9/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999827    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.311137011s: [/var/lib/containers/storage/overlay/b8b56f59390f731005dd4f5e142899c74da144d464f61fbeea461b527f3dfc3d/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/prom-label-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999837    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.311162677s: [/var/lib/containers/storage/overlay/a6efa294561d8867fca81b5aa12f5b9f4ce0955f3122e0d2c68924d7a6db53f8/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy-rules/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999851    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.311176192s: [/var/lib/containers/storage/overlay/2baaae2b22b3f8e778df214acf8cc253c7a7bfa34a8e2a49f4c49e86ff018e9a/diff /var/log/pods/openshift-console_downloads-6dd789c8b7-pgmzg_6e78d6c7-c325-48f2-af90-804dade84d8f/download-server/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999861    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m18.835286891s: [/var/lib/containers/storage/overlay/e7bd185adbe2a5f6c8ee6467194d0c245d24e52db000b36e4df509e6ff6b0e8a/diff /var/log/pods/openshift-ingress_e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq_778b9a36-6a7b-4251-a91a-999e358d8c5c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999871    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.315556067s: [/var/lib/containers/storage/overlay/38e2db64053a75d5e267a571b5c8029e880b65c662fc935cba8b4b78f3031bad/diff /var/log/pods/openshift-ingress_e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc_ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999895    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m18.384436355s: [/var/lib/containers/storage/overlay/35ea380649cc6d0f8dc0e907ecb9ef66961662f67283708985d685f75f06497f/diff /var/log/pods/openshift-ingress_e2e-worker-w1-120394-openshift-default-6459698f85-j24dq_acd44782-84e1-411a-9a1c-6d21456733bb/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999895    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.127686203s: [/var/lib/containers/storage/overlay/a55dca0d0429c95641f315985403a890eb29899befd46ff107f67affa1d6343d/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw_5b0ff069-2a17-489e-ab3d-6d5461bffbf4/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999912    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.11372993s: [/var/lib/containers/storage/overlay/48d5c82df7e650a089ea0fa91ed0bd4c1b6a9b1f380d8d8db83882fc5cc1bf60/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-driver/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999926    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.113739215s: [/var/lib/containers/storage/overlay/d5d50d36d455b3124bba1a4682c6b556a344e4c0f72561f6ab14472bbe327aa9/diff /var/log/pods/openshift-multus_multus-additional-cni-plugins-ngkbz_c2d7ba52-9a2b-4871-aca0-3c4e1934c572/kube-multus-additional-cni-plugins/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999940    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.113752299s: [/var/lib/containers/storage/overlay/ec5326cd4c57e5289c85be1ed540648b29113e91e4b3f4a2026c4c66a8e87d65/diff /var/log/pods/openshift-ingress-canary_ingress-canary-n4r8t_ba5cbec9-7649-462a-b110-11ebcc5d6d78/serve-healthcheck-canary/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999948    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.113693701s: [/var/lib/containers/storage/overlay/613c01497ad5e2672a318b68ad56355e93f5e704f0918037c43a2d38d967a924/diff /var/log/pods/openshift-monitoring_telemeter-client-79c994cf5c-ckj5h_e065a5c3-95fd-4ac8-a287-8e1d20219201/reload/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999971    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.31444218s: [/var/lib/containers/storage/overlay/6711a046696ca85a051e460f8eca0f62ee6736867b06bec3453afa502df3ff47/diff /var/log/pods/cert-manager_cert-manager-cainjector-5c57bbd4fc-9mjv9_9fe1660e-3733-4c7f-bc5c-3820d84aa143/cert-manager-cainjector/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999976    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.314443811s: [/var/lib/containers/storage/overlay/968e208b4860faed3d3e26aa71e41adb5c447ce22ac3bb6cd22690346d2dfb5c/diff /var/log/pods/e2e-models-e2e-worker-w3-5eedab_distinct-w3-5eedab-kserve-856b8c7458-k2lgz_0874d0e8-5883-477d-b64a-d455934cfc90/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:56.999993    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.568480456s: [/var/lib/containers/storage/overlay/b8c1da6792996b2b843a3227a1547b84d7bccce31ef5f330614abc41c7d98055/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75_7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f/maas-api/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000001    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.127799019s: [/var/lib/containers/storage/overlay/d002b0ce8a03e77141888fb5a4a50972cf6108b7e5e59c9375aa48cad9ced5af/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx_d837dde2-3939-4400-8ea7-fee58c162983/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000018    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.113400226s: [/var/lib/containers/storage/overlay/48a8a99b5aa38b5a1f8af93f2cf91f2afe257cea7b9b3fc9ad86c404aea4cfa8/diff /var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000026    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.473703022s: [/var/lib/containers/storage/overlay/8e743a1c0c7cc9ef45cd9f86c51bdd235879c512aca4840e3a446be2f316bb6e/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr_f4ee5c50-5738-496a-b5ef-c86ead76e0f6/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000042    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.113394954s: [/var/lib/containers/storage/overlay/df02c5a738481c502ee0fbb85ca8fb2c47529fde2b1bfbb73efe51cb364c310b/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/ovn-controller/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000056    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.358766125s: [/var/lib/containers/storage/overlay/7c82bb52ad040de109f8f96e4fde961296a9a03788caea2ee7088d4b4b9b113b/diff /var/log/pods/openshift-ingress_e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt_82ba53b4-e91a-4299-9498-f5abcd34576b/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000065    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.127860847s: [/var/lib/containers/storage/overlay/ad9f3bb11ff9cab22055aad521309605cbe31ca829c1820e58022c92cded277f/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6_1c190514-ba33-4844-a9ec-e4bb47b5f706/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000041    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000336    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000080    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.113582526s: [/var/lib/containers/storage/overlay/0ba4937b318aff26f5d04de663e3cbc5fb91d34ecbaf5c1a70e275fa82d80f53/diff /var/log/pods/openshift-ingress_e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z_d4161075-440d-4373-b878-57c77612c6d2/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000094    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.113410762s: [/var/lib/containers/storage/overlay/508796c136616a6a941f39c30320d167ca9f8cc36fab3ce4085e5e6164b8c836/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr_e141d845-db9d-4f94-b05e-cfb7848e8fef/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000098    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.113407553s: [/var/lib/containers/storage/overlay/903a7a7c7002a9034e9b89db1ce800ddd9a85962e7522211d8fa957ab69f9ec0/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-m89vg_9257d59b-6aaf-423e-b7ed-2e64dbdfa535/kube-rbac-proxy-main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000116    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.315807771s: [/var/lib/containers/storage/overlay/07e8324700651cc1b533fd9bd9bd56d25f48721c5261994354ea99a5533a21f1/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx_38977edb-8612-48c6-a6c6-1714a50a9e4f/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000133    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.112656755s: [/var/lib/containers/storage/overlay/45ab15387881b8da1f41ebe3c7b88bd293020499c9cacdb6a8c2b4a9b2ed2fb6/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg_b8847684-5676-4253-850f-9f2b2c42ea0c/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000137    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.707219628s: [/var/lib/containers/storage/overlay/acc66a71ede97842cda782e51e0adbea72945cd7fa0b1af2fc4160e227b7062f/diff /var/log/pods/openshift-ingress_e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5_0996b2fa-40e8-469c-9ace-75458691fa25/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000151    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.330535684s: [/var/lib/containers/storage/overlay/a9f451844d697710f8715a06cae580171d3d191b1c96eaef66ef0f4efdbe4e6a/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v_6ebd3467-841a-4331-b149-8acf6e60c12c/maas-api/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000156    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.113476669s: [/var/lib/containers/storage/overlay/67b48a8227291881bea04d0f599da8269338873aa7b51eb6d44f387b34a084de/diff /var/log/pods/e2e-models-e2e-worker-w5-863b87_simulator-w5-863b87-kserve-bd54699cd-7g2mv_76d3e197-83a0-4793-8593-0b47b55a9fc4/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000531    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.057299861s: [/var/lib/containers/storage/overlay/8c0ca4c9e4e4e93687de6c7e703ba32222bc3330b9cf5dbd83a5cbdfbc1eb1a4/diff /var/log/pods/openshift-ingress_e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r_67b961eb-ab19-4d8d-bb2a-db7c94a93436/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000167    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.11247063s: [/var/lib/containers/storage/overlay/37c279f7138b64cf848e046dd0159412d2f60a6e3f561819a069d84c7835dd5e/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/ovnkube-controller/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000172    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.112472419s: [/var/lib/containers/storage/overlay/68adcccdea3eba205d654c160d713ccd9fe3f136335e3c8c4fd9800b1d4da9c1/diff /var/log/pods/openshift-network-diagnostics_network-check-target-nttxj_a974c7e3-bf97-4575-bec1-c09fcd018f0d/network-check-target-container/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:12.706653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000185    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.112469877s: [/var/lib/containers/storage/overlay/b7702443b08031bf70dfb72c9da1f832917de0a24b9b53011dbe509513501266/diff /var/log/pods/kube-system_global-pull-secret-syncer-mmhd7_8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d/global-pull-secret-syncer/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000189    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.112410784s: [/var/lib/containers/storage/overlay/1b551b04f397d2a17bb4b72d2a9e13ea7b255a0c2887bd5f17c4e17a3c5d3890/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/kube-rbac-proxy-node/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000554    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.057329956s: [/var/lib/containers/storage/overlay/d0ef0428e1dc75ad127901851f1b2fc725d21cd1b094959f49dcd7d557ab4f1a/diff /var/log/pods/openshift-ingress_e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths_5775b5a6-6407-46f4-84ef-96c708d7934b/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000202    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.311529909s: [/var/lib/containers/storage/overlay/d5f8aa2b7ec7f078e5d3587bf3acd312cea2db5330296f8eeaf6747bb8ecb4af/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6_60c0e838-5c79-4bf7-a7c6-87903ae8bd8c/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000206    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.113852255s: [/var/lib/containers/storage/overlay/e68ebf6860926a21a0cbfecd410ae12a4c05fb62dae7bcdbf219e8a050cbd79d/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8_e4c1132f-05b5-4664-84a2-ece57727b12a/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000219    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.314685773s: [/var/lib/containers/storage/overlay/1d99eb2b6b4423a5208899af1d81194c5531b55c6df383327a20817e35f00610/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89_8c60dfaa-dabe-4726-91b5-165443ff3971/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000223    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.11224899s: [/var/lib/containers/storage/overlay/d0014aabc7632aacf8f230d1d606750084728ea64d88ba0a6c129361bdd715f8/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/thanos-sidecar/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000237    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.112259001s: [/var/lib/containers/storage/overlay/39036cdcb7e7029ac2d15241aa50d868ff1a012fe62623885c01ed6f7528dbce/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000239    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.112260743s: [/var/lib/containers/storage/overlay/afcf005d130d4b0994316011111d3ba89bede3db3572360e1b8cddd15c5a4284/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000591    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000257    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.533141548s: [/var/lib/containers/storage/overlay/801517f9d9b867bed7e9f37f0f99b4b285f8fe58d0c6fe3b4e406c4b66eb885f/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g_c0ff0cbb-4acf-4d63-9579-6aeac659b2ea/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000259    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.075106652s: [/var/lib/containers/storage/overlay/05b2f9768ded3bb89850b28832c2c3463a2ffa7b9eab99ff39d4e91999d29770/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-tdx82_0f2089f4-8ec7-45ce-a2df-0bf024b4158f/discovery/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000885    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.040023648s: [/var/lib/containers/storage/overlay/0a6251ac8edb01e4a1a495ea3104a4be3776a4bde7a9167c9216fe870b0b5a1d/diff /var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-87.ec2.internal_fbfcd267e5bdda32002f5b72a55d4954/haproxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000274    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.311746438s: [/var/lib/containers/storage/overlay/8e3943818fa8cc19ef6f82354867dfbe856e505e34cf4a7d05bc429843ee18e5/diff /var/log/pods/kuadrant-system_authorino-6f5b7cd5c7-brdvs_040ec9bf-319e-4bae-b233-4e4e7e74a379/authorino/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000292    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.075046485s: [/var/lib/containers/storage/overlay/b569994ab3a7fd09901b9179b5a8ed81afe826237d0aeefc81429585cefd651a/diff /var/log/pods/e2e-models-e2e-worker-w1-120394_simulator-w1-120394-kserve-6979bf4f4f-scg5b_5734524d-67b2-4e24-a836-3ac078d5a271/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000293    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 11.571802651s: [/var/lib/containers/storage/overlay/fe9920e5a7065775346550e5358cc11abb4093be2682b270b828978d1fc3bdbb/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000308    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.112760311s: [/var/lib/containers/storage/overlay/3b4eb99b514a85975e9e184eb65b4fc878deddd7205c795539ae51ac4de5caab/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj_bca2dabc-94e2-4875-ba3f-fa9332afe8dc/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000313    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.075061282s: [/var/lib/containers/storage/overlay/2850c96a313d433f759e56135d890d3c3b9d4452f3e507896aa256d912e71270/diff /var/log/pods/openshift-cluster-node-tuning-operator_tuned-n9fk9_f4b72e7f-cc4c-4662-a923-c184069aa118/tuned/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000328    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.316018206s: [/var/lib/containers/storage/overlay/1d679dc376e8b4b61ee5b35c347f887a08cc74c2c68502794ffcfd9744b157de/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq_5e54b0f0-5c3c-4441-a2a6-b452b9715e2e/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000940    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.040040195s: [/var/lib/containers/storage/overlay/87c11e77a69c1256c563023edeadf4b64a91805f1340898de6ce44397e54a787/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000331    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.07507031s: [/var/lib/containers/storage/overlay/69e9e5b9009b44f81cb13f7e89161acc514055b4129ec8240b07d4f06c225a45/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-node-driver-registrar/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000345    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.075077845s: [/var/lib/containers/storage/overlay/065839dc8c2c6a550d10b437944e397361cfc89bf9ccb63c18a6d705e7669644/diff /var/log/pods/openshift-multus_multus-7qzkz_f26fab1e-e6d4-4caa-bef6-81a09d78c0de/kube-multus/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000347    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.075052522s: [/var/lib/containers/storage/overlay/f04694bfe7aed9c3a051f679dbed90340f776045c06d3bc39068d91ebc64b1d9/diff /var/log/pods/openshift-dns_dns-default-zmrdt_f59eba5d-a500-462e-9ecd-4ef9e553f873/dns/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000365    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.114241594s: [/var/lib/containers/storage/overlay/ffc640e61d9f2b8dc312d9f010b835c9e91e982e4f9cff0f120f887c46ef8f3a/diff /var/log/pods/openshift-ingress_e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb_551602c4-254b-4532-8961-a98bf0f81089/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000366    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.075040607s: [/var/lib/containers/storage/overlay/6da4544d5bf904910dc0072ef528e9149345d962762f71ecb9b0ce270fe91dc2/diff /var/log/pods/openshift-insights_insights-runtime-extractor-6bfcf_da9f1b66-0e1b-4c7e-a947-a897c00ccc19/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000384    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.075054012s: [/var/lib/containers/storage/overlay/089a992d10746017d3c97ab20d1ddf9b955c8babb89ca5a1613bd5a1985f773b/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-m89vg_9257d59b-6aaf-423e-b7ed-2e64dbdfa535/openshift-state-metrics/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000384    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.075056128s: [/var/lib/containers/storage/overlay/38cc698afa3d0a22ee2e64e3fa8b08e084747b51f63934648a8d8162d387223b/diff /var/log/pods/openshift-insights_insights-runtime-extractor-6bfcf_da9f1b66-0e1b-4c7e-a947-a897c00ccc19/exporter/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000401    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.075049249s: [/var/lib/containers/storage/overlay/961cc1369cc5c2b2a8db1c37483931e3d284d8bf3daf0ea781ee526fa377a6a2/diff /var/log/pods/openshift-monitoring_metrics-server-7d987f75-ntp98_18d2ae0a-bf65-4aee-8730-093e00b54a24/metrics-server/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000403    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.075052282s: [/var/lib/containers/storage/overlay/4c8d263b1f9b1e5f7f61488e19a80e3bdfd0a74fc133c4635b7c83f9383972d5/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm_dbd34da0-9b90-458d-9581-ed9caf3daa11/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000500    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.112952761s: [/var/lib/containers/storage/overlay/fa8831aafda2d92f0ce3f59ac079b571f74f757924d7c3247d164db1d6711b3c/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j_e5ae7b61-cd5f-4a38-b038-28aaa6991eb7/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000515    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.075256102s: [/var/lib/containers/storage/overlay/6938a4b799445ead8836745535273347f3c908c027c27cae3317fe23a343f2cc/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/northd/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000531    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.075282499s: [/var/lib/containers/storage/overlay/cb4bed28be09871a074b3642832880027d77099e69a0ae8241aeb94dab27fdaf/diff /var/log/pods/openshift-image-registry_node-ca-lhwgm_00b87b36-9ce2-480a-9924-22b33256fe1e/node-ca/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000582    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.11391258s: [/var/lib/containers/storage/overlay/c054c6b27f869c3385aeca4dc9bbc0f3c6a68aec5c08b24d9bfc20cfd651da17/diff /var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-xc9kg_3b0e614e-16b1-4e28-bea6-4b7b9a62b177/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000599    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.057314523s: [/var/lib/containers/storage/overlay/715bfa0d02500a7607ee5ba949c9655d2d098909dc187d3fa64c431aa173e55c/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb_cc682b81-5f9e-4cd1-98ae-d32e89878ea3/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000621    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.057332604s: [/var/lib/containers/storage/overlay/04144c7db0ef8494e8777526c23ed619a940be7d263e9bd174a3253037d6d77a/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d_8cba3353-fa51-41d3-acb4-a3cfbaae1daa/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000638    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.057346829s: [/var/lib/containers/storage/overlay/61dba17c91911e3fe21e3760810cb0fdaa44600557f817d2a6d5f9e5af198421/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn_3d96e066-10c0-4fe2-9e24-ffd6195b22af/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000662    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 20.316346427s: [/var/lib/containers/storage/overlay/4832bee5d30fb081704a3c96e43d22d7214d83e63b2bbc7c3a50de9c878d8550/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46_806cc8ce-f608-4975-94dd-5edade11fc8c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000679    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.040160865s: [/var/lib/containers/storage/overlay/5c9ea955e524e35aff9b30ed78d97e370413ede7d4597334da2209e4500672c6/diff /var/log/pods/openshift-ingress_e2e-inf-a-872354-openshift-default-59df9df7-vnmfl_3c8d9150-66d1-4be8-8bca-3782fb0eac36/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000694    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.040155504s: [/var/lib/containers/storage/overlay/c032743b3f5164f05268a34d27f7a38a2df55f102c647c344a128377fc34aefd/diff /var/log/pods/openshift-ingress_e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx_81fd82a8-2bbf-43c8-962c-f8384287dc1f/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000709    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.040167268s: [/var/lib/containers/storage/overlay/98ff010f77e7d96f5843be7a3a27066c1c197ffa6885b42fa9f473d20af37b2f/diff /var/log/pods/openshift-ingress_e2e-inf-a-872354-openshift-default-59df9df7-v5qhw_32802945-5e87-4339-a7eb-33212aab7ba5/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000723    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.11406849s: [/var/lib/containers/storage/overlay/83f8cdd611cedca899243d5a228c934d7754b14823c1219db9a0f8ced5da014e/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-5bqck_280a3e6c-c540-4c66-b963-c678ddca6e40/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000740    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.03998662s: [/var/lib/containers/storage/overlay/25ec8cf788a39d4012f5db549bcc3d440d265141a6b16d25ff0974d692c4631b/diff /var/log/pods/openshift-ingress_e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8_6aa41ecd-4a93-4b17-8e84-3f3c018c4d21/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000755    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.040001421s: [/var/lib/containers/storage/overlay/861acfcf2321b99e26e85c3154bd9202cc22f4d667e5041f027bd7d24464ecb9/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-liveness-probe/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000771    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.039993717s: [/var/lib/containers/storage/overlay/48b947c743d65c0ae34868fbb47aeb890dde93159790d378edd81be829b75cfb/diff /var/log/pods/openshift-dns_node-resolver-hhsrb_ce8b7505-62a5-42f1-b432-d77200c21d2d/dns-node-resolver/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000788    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.112853658s: [/var/lib/containers/storage/overlay/6890c8f8933a1e87b2636594da397d70dd586fa37081a11d1320e3554116bda7/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-vrvf2_07f874bb-122b-4794-ab59-736495e5cfdc/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000808    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.040027563s: [/var/lib/containers/storage/overlay/1550c3e2e2a45dcfd84afdd33aa3690aa5dcb9b9475cb07cc27b0e0b34c41bb3/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/sbdb/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000864    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.04006963s: [/var/lib/containers/storage/overlay/e34d2a2f5077d6161aee9054ac97bc88dad0afe4dadc0770608b95d5aacf2f11/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/ovn-acl-logging/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000923    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.040022902s: [/var/lib/containers/storage/overlay/38f6c3a3bf1761682b345c03cb02c31f484b894296aec5d499ce5ae2e5601af5/diff /var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal_375359043409c490a8ed0d21b8afeaf5/kube-rbac-proxy-crio/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.000955    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.040002251s: [/var/lib/containers/storage/overlay/1cfef9a8ec66cc7e72a2cc8f5db5f517a05984ada8feb9b6fcab0c85033fb6e5/diff /var/log/pods/openshift-multus_network-metrics-daemon-tk8ht_dd733778-7002-4a26-a4e8-4525d8f6bee9/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.001206    4456 patch_prober.go:29] interesting pod/image-registry-59767d8795-7vwkt container/registry namespace/openshift-image-registry: Liveness probe status=failure output="Get \"https://10.134.0.13:5000/healthz\": context deadline exceeded" start-of-body=
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.001240    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" podUID="0db36d92-4712-4338-958a-ebabf2c137ba" containerName="registry" probeResult="failure" output="Get \"https://10.134.0.13:5000/healthz\": context deadline exceeded"
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.001549    4456 trace.go:236] Trace[1894923768]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" (02-Nov-2026 18:04:38.912) (total time: 18088ms):
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: Trace[1894923768]: [18.088759078s] [18.088759078s] END
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.001587    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.115017475s: [/var/lib/containers/storage/overlay/4aaa9c0327050d64edf543a80049fd2fd46339540b39fe37be6c43a26d8f4b91/diff /var/log/pods/openshift-ingress_e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh_ef6443b3-b452-4868-9cde-f762cf522284/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.001937    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.476175464s: [/var/lib/containers/storage/overlay/0f64d44e017ec70c0f08927d7a5b5315efaaa27e26b52cb55a2d48db8ff58cb5/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4_18a32a7b-66fa-4f34-ab36-2beb0169e430/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.003178    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.077930915s: [/var/lib/containers/storage/overlay/d09ba0e972daf0879be4ba9795c180ae5ce98f9c793ff54605297f1c7dd5eb3d/diff /var/log/pods/openshift-ingress_e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88_cb09170f-1b12-4a26-8e5b-c5e2c2964a4d/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.004030    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 22.205884043s: [/var/lib/containers/storage/overlay/a143e700ec1dc2f1d0157b0b73e4385b467977554b146f0849b335957d53870f/diff /var/log/pods/openshift-ingress_e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7_73849b3d-673e-41a8-aff1-db9f0a9cce85/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.004084    4456 trace.go:236] Trace[1498827803]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" (02-Nov-2026 18:04:38.905) (total time: 18098ms):
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: Trace[1498827803]: [18.098979808s] [18.098979808s] END
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.004283    4456 trace.go:236] Trace[534086472]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" (02-Nov-2026 18:04:38.615) (total time: 18388ms):
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: Trace[534086472]: [18.38836789s] [18.38836789s] END
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.005227    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.117288687s: [/var/lib/containers/storage/overlay/985ba12eb48566836f0fcad469e6eeb2711b77cf3eca5b054713e9dd9c612af3/diff /var/log/pods/openshift-ingress_kube-auth-proxy-78648788bf-npsk7_347c8613-0791-494e-83e3-faded5593d88/kube-auth-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.005972    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.119324213s: [/var/lib/containers/storage/overlay/1488f5a27e0831e0c877985c837f753313a612723997edaa573902b65e840bd5/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f_5ce3335c-36e6-4018-a264-5b79085f09f3/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.006883    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.118969807s: [/var/lib/containers/storage/overlay/3e001268063ff1680111524029f98c7759f3a4feccf0afcf7064a2ca68c37a83/diff /var/log/pods/e2e-models-e2e-worker-w0-069fa5_premium-w0-069fa5-kserve-699ddf764c-xmspr_9e85cf3f-eb7a-452b-b716-26df5b391065/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.008983    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.009021    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.010003    4456 patch_prober.go:29] interesting pod/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.010039    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.69:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.010351    4456 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.010382    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.010704    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.010739    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.011620    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.011686    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.011651    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": dial tcp 10.134.0.28:8081: connect: connection refused" start-of-body=
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.012310    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": dial tcp 10.134.0.28:8081: connect: connection refused"
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.012955    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" podUID="44b66f41-99ae-446b-a50a-2ec0b71563a4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.189:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.014231    4456 scope.go:122] "RemoveContainer" containerID="e1922175d5513058013bf3b85e2a3221900f1fde3d1509b5419561d2ef700c83"
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:04:57.015914    4456 kubelet.go:2545] "Skipping pod synchronization" err="container runtime is down"
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.018302    4456 patch_prober.go:29] interesting pod/image-registry-59767d8795-7vwkt container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.134.0.13:5000/healthz\": context deadline exceeded" start-of-body=
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.018334    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" podUID="0db36d92-4712-4338-958a-ebabf2c137ba" containerName="registry" probeResult="failure" output="Get \"https://10.134.0.13:5000/healthz\": context deadline exceeded"
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.020443    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.134.0.28:8081/healthz\": dial tcp 10.134.0.28:8081: connect: connection refused" start-of-body=
Nov 02 18:05:13.223578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.020476    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/healthz\": dial tcp 10.134.0.28:8081: connect: connection refused"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.022985    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2" podUID="49238ac8-57fa-412a-88bd-681364a0b455" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.230:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.023944    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.023976    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.024233    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": context deadline exceeded"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.025259    4456 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-tdx82 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.025288    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" podUID="0f2089f4-8ec7-45ce-a2df-0bf024b4158f" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.025366    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.025397    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.028187    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.028220    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.029914    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.030651    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.105339495s: [/var/lib/containers/storage/overlay/1a5df3100db9dd3cfd75b7f681887de0a6a1fe77ee9e7827781cfdc3b41a2d4d/diff /var/log/pods/openshift-ingress_e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck_694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.035683    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.092370688s: [/var/lib/containers/storage/overlay/b72b6eb4d0d5d620757c710e468b44a22e782ad83f727a0ccfc574805e34069a/diff /var/log/pods/cert-manager_cert-manager-597b95cb84-hr8vz_e73be583-2a26-4379-8169-0a6b8784a11b/cert-manager-controller/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.036053    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j" podUID="e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.037474    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.094225082s: [/var/lib/containers/storage/overlay/79ff9c7f890e079eb19a8a8060e78ed7f0f7644efb29ded7f8d223b309ccdf2d/diff /var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k_14d43510-cfbb-4ddc-88cb-ccd4cb3a543d/main/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.037610    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.170:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.039566    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.096338247s: [/var/lib/containers/storage/overlay/8df838514665e6e27e507b914dcdc2f57a13508b4557b623dfa7fb79412fd9b0/diff /var/log/pods/openshift-ingress_e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f_e5167f1d-3718-4c96-bf19-1cdc1ab73207/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.039627    4456 patch_prober.go:29] interesting pod/dns-default-zmrdt container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.134.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.039658    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-zmrdt" podUID="f59eba5d-a500-462e-9ecd-4ef9e553f873" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.042509    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": context deadline exceeded"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.043662    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.082908307s: [/var/lib/containers/storage/overlay/a2aa71b482739c7b0a0cc470b85d5d8947410f51100d4866f063fe123ede7d3e/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr_22b556c1-112b-49d4-932c-0143e912a54e/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.044407    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.045412    4456 trace.go:236] Trace[1615776573]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2" (02-Nov-2026 18:04:38.917) (total time: 18128ms):
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: Trace[1615776573]: [18.128191749s] [18.128191749s] END
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.049651    4456 trace.go:236] Trace[1090172340]: "Calculate volume metrics of nginx-conf for pod kuadrant-system/kuadrant-console-plugin-66c4fb575d-c8vr7" (02-Nov-2026 18:04:38.564) (total time: 18485ms):
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: Trace[1090172340]: [18.485312057s] [18.485312057s] END
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.049710    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.052935    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.054422    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded" start-of-body=
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.054455    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.058325    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.059107    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.061951    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" podUID="22b556c1-112b-49d4-932c-0143e912a54e" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.062656    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-mxsjq" podUID="367d8d85-a5e8-462d-b1d7-91a01e62b726" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.064345    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" podUID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.064436    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.064635    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.064938    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.065114    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5" podUID="02a2dbe6-baeb-40c4-be0f-835239be331f" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.066119    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" podUID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.066555    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.066940    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" podUID="d837dde2-3939-4400-8ea7-fee58c162983" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.067791    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" podUID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.067957    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.068259    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.068714    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" podUID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.070793    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.080912    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" podUID="e141d845-db9d-4f94-b05e-cfb7848e8fef" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.180:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.084946    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.129:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.084995    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.086022    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" podUID="e4c1132f-05b5-4664-84a2-ece57727b12a" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.088895    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.163528706s: [/var/lib/containers/storage/overlay/be0a87d2eccccae8e85688b90a61b88439e6fd706c76be4ba52b6f95c4beac73/diff /var/log/pods/openshift-ingress_e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz_408afe2d-75a0-4537-b58c-9ab31f161685/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.636337    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.693083891s: [/var/lib/containers/storage/overlay/8fe16763de809dbbc885c5aa483909cdeb3f142a735700e1976d0912af066878/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2_49238ac8-57fa-412a-88bd-681364a0b455/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.636605    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.750025472s: [/var/lib/containers/storage/overlay/b2355f8f3cbf78f8aa7f79c1f2cfd6f7039b2e313a3bb74da28ad5f343642d01/diff /var/log/pods/openshift-jobset-operator_jobset-controller-manager-77cfb5f94-gfbr9_6957a218-62fc-4dd8-892e-4477b08c046f/manager/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.639762    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.679326117s: [/var/lib/containers/storage/overlay/2451471e892c455bc0429337eed2450e847bf657dfb5b21286782c82632452ed/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz_b5d45c38-1cef-4aa9-a519-98aed00e8727/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.722481    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 25.487868284s: [/var/lib/containers/storage/overlay/833351d2ebd14fa986d0203e8af3254003326af193a6655f85b7084ad760a6ec/diff /var/log/pods/openshift-ingress_e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf_999df778-cd9b-4b41-852d-4661a7326483/istio-proxy/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.722505    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m6.405695988s: [/var/lib/containers/storage/overlay/7840ca7106125a4b22fa575aa093373b243546e0cef5c452d1dd3a71532c03ea/diff /var/log/pods/openshift-ingress_e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt_856c2dca-bad2-4f62-a536-1ca0e09f0a12/istio-proxy/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.774529    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.775722    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.775745    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.777338    4456 patch_prober.go:29] interesting pod/metrics-server-7d987f75-ntp98 container/metrics-server namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.16:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.777366    4456 patch_prober.go:29] interesting pod/metrics-server-7d987f75-ntp98 container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.16:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.777403    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/metrics-server-7d987f75-ntp98" podUID="18d2ae0a-bf65-4aee-8730-093e00b54a24" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.16:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.777407    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-7d987f75-ntp98" podUID="18d2ae0a-bf65-4aee-8730-093e00b54a24" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.16:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.796711    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.796724    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.15:9091/-/healthy\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.796777    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/healthy\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.798887    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.798899    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.798922    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.798991    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.799180    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.799336    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.802332    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.806707    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.806720    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.806749    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.806796    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.807264    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.807287    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.814943    4456 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-pgmzg container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.814968    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.815917    4456 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-pgmzg container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.815954    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.815987    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.816107    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.816297    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"]
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.816330    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 18:05:13.295535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.816372    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.816400    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.816429    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.816645    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": dial tcp 10.134.0.28:8081: connect: connection refused" start-of-body=
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.816675    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": dial tcp 10.134.0.28:8081: connect: connection refused"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.836870    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.839039    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.844543    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" podUID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.182:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.845528    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.846919    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.849383    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.877915    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:57.884965    4456 logging.go:55] [core] [Channel #5928 SubChannel #5929]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.182:50052", ServerName: "10.134.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:45678->10.134.0.182:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.888926    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.888939    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.942154    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.975099    4456 ???:1] "http: TLS handshake error from 10.0.141.87:47370: EOF"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:57.994356    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.001116    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.001138    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.005790    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.005842    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.013901    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" podUID="44b66f41-99ae-446b-a50a-2ec0b71563a4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.189:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.023178    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2" podUID="49238ac8-57fa-412a-88bd-681364a0b455" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.230:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.030383    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.42:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.036625    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j" podUID="e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.44:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.038201    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.170:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.045261    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.053154    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.134:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.054954    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.054994    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.055004    4456 logging.go:55] [core] [Channel #5931 SubChannel #5932]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.189:50052", ServerName: "10.134.0.189:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:32926->10.134.0.189:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.058460    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.062574    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" podUID="22b556c1-112b-49d4-932c-0143e912a54e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.186:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.063963    4456 logging.go:55] [core] [Channel #5934 SubChannel #5935]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.230:50052", ServerName: "10.134.0.230:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:56180->10.134.0.230:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.064678    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" podUID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.172:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.064712    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.200:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.064791    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.176:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.065772    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5" podUID="02a2dbe6-baeb-40c4-be0f-835239be331f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.135:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.065781    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.067003    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.173:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.067038    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" podUID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.067139    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" podUID="d837dde2-3939-4400-8ea7-fee58c162983" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.116:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.068260    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" podUID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.183:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.068369    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.43:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.068421    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.175:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.069599    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" podUID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.190:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.070969    4456 logging.go:55] [core] [Channel #5937 SubChannel #5938]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.42:50052", ServerName: "10.134.0.42:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:59144->10.134.0.42:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.071033    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.187:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.076959    4456 logging.go:55] [core] [Channel #5940 SubChannel #5941]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.44:50052", ServerName: "10.134.0.44:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:41398->10.134.0.44:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.078947    4456 logging.go:55] [core] [Channel #5943 SubChannel #5944]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.170:50052", ServerName: "10.134.0.170:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:59908->10.134.0.170:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.082071    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" podUID="e141d845-db9d-4f94-b05e-cfb7848e8fef" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.180:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.085195    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.201:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.085201    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.129:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.086299    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" podUID="e4c1132f-05b5-4664-84a2-ece57727b12a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.122:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.091917    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.093966    4456 logging.go:55] [core] [Channel #5946 SubChannel #5947]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.134:50052", ServerName: "10.134.0.134:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:34476->10.134.0.134:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.104010    4456 logging.go:55] [core] [Channel #5949 SubChannel #5950]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.186:50052", ServerName: "10.134.0.186:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:38426->10.134.0.186:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.104964    4456 logging.go:55] [core] [Channel #5952 SubChannel #5954]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.172:50052", ServerName: "10.134.0.172:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:51594->10.134.0.172:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.104979    4456 logging.go:55] [core] [Channel #5953 SubChannel #5956]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.200:50052", ServerName: "10.134.0.200:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:57250->10.134.0.200:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.106963    4456 logging.go:55] [core] [Channel #5955 SubChannel #5957]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.176:50052", ServerName: "10.134.0.176:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:46466->10.134.0.176:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.106968    4456 logging.go:55] [core] [Channel #5959 SubChannel #5963]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.135:50052", ServerName: "10.134.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:44256->10.134.0.135:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.106996    4456 logging.go:55] [core] [Channel #5958 SubChannel #5962]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.171:50052", ServerName: "10.134.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:45578->10.134.0.171:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.107946    4456 logging.go:55] [core] [Channel #5971 SubChannel #5972]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.116:50052", ServerName: "10.134.0.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:60864->10.134.0.116:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.107948    4456 logging.go:55] [core] [Channel #5969 SubChannel #5970]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.173:50052", ServerName: "10.134.0.173:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:34678->10.134.0.173:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.107989    4456 logging.go:55] [core] [Channel #5967 SubChannel #5968]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.45:50052", ServerName: "10.134.0.45:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:37508->10.134.0.45:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.109980    4456 logging.go:55] [core] [Channel #5976 SubChannel #5978]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.183:50052", ServerName: "10.134.0.183:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:58474->10.134.0.183:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.109990    4456 logging.go:55] [core] [Channel #5979 SubChannel #5980]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.175:50052", ServerName: "10.134.0.175:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:51772->10.134.0.175:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.110023    4456 logging.go:55] [core] [Channel #5981 SubChannel #5984]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.190:50052", ServerName: "10.134.0.190:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:54930->10.134.0.190:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.110062    4456 logging.go:55] [core] [Channel #5977 SubChannel #5982]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.43:50052", ServerName: "10.134.0.43:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:60884->10.134.0.43:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.111949    4456 logging.go:55] [core] [Channel #5983 SubChannel #5986]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.187:50052", ServerName: "10.134.0.187:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:36670->10.134.0.187:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.122948    4456 logging.go:55] [core] [Channel #5985 SubChannel #5992]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.180:50052", ServerName: "10.134.0.180:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:52750->10.134.0.180:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.125971    4456 logging.go:55] [core] [Channel #5993 SubChannel #5995]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.201:50052", ServerName: "10.134.0.201:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:33496->10.134.0.201:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.125975    4456 logging.go:55] [core] [Channel #5991 SubChannel #5994]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.129:50052", ServerName: "10.134.0.129:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:35802->10.134.0.129:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:04:58.126962    4456 logging.go:55] [core] [Channel #6000 SubChannel #6001]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.122:50052", ServerName: "10.134.0.122:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:43426->10.134.0.122:50052: use of closed network connection"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.419279    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 50.392671374s: [/var/lib/containers/storage/overlay/48c65928388ea29afd8ce97d9eec05ed39a6fd3da709a73d2174493d7947d9f4/diff /var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-c8vr7_08c9aa32-dadc-4112-b608-4cc64f56295c/kuadrant-console-plugin/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.776167    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.776167    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.799731    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.799779    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": context deadline exceeded"
Nov 02 18:05:13.399590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.799858    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.799780    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.807602    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.807602    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.839597    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.847833    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:58.994876    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:59.002691    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:59.005424    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:59.404120    4456 trace.go:236] Trace[895931047]: "Calculate volume metrics of ca-trust-extracted for pod openshift-image-registry/image-registry-59767d8795-7vwkt" (02-Nov-2026 18:04:56.900) (total time: 2503ms):
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: Trace[895931047]: [2.503963367s] [2.503963367s] END
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:59.775877    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:59.784681    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="83deda5b-6ef6-4a66-a76c-888303e6e834" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:59.843937    4456 patch_prober.go:29] interesting pod/dns-default-zmrdt container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:59.844001    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-zmrdt" podUID="f59eba5d-a500-462e-9ecd-4ef9e553f873" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:04:59.995790    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:00.001283    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": context deadline exceeded"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:00.043437    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:00.064913    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:00.064965    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:00.065889    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:00.065921    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:00.101029    4456 trace.go:236] Trace[1965685164]: "iptables ChainExists" (02-Nov-2026 18:04:36.671) (total time: 23429ms):
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: Trace[1965685164]: [23.429639218s] [23.429639218s] END
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:00.101061    4456 trace.go:236] Trace[1442258970]: "iptables ChainExists" (02-Nov-2026 18:04:36.683) (total time: 23417ms):
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: Trace[1442258970]: [23.417372643s] [23.417372643s] END
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:00.268691    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="83deda5b-6ef6-4a66-a76c-888303e6e834" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:00.848865    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:00.848934    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:01.351540    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" podUID="c86c4f40-2978-40f0-bfc1-c3cfffca0d8c" containerName="sbdb" probeResult="failure" output="command timed out"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:01.351557    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" podUID="c86c4f40-2978-40f0-bfc1-c3cfffca0d8c" containerName="nbdb" probeResult="failure" output="command timed out"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:01.439197    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:01.653944    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:01.668094    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:05:01.867739    4456 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="4.051s"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:01.867780    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:01.867836    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:01.990978    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.002136    4456 patch_prober.go:29] interesting pod/image-registry-59767d8795-7vwkt container/registry namespace/openshift-image-registry: Liveness probe status=failure output="Get \"https://10.134.0.13:5000/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.002191    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" podUID="0db36d92-4712-4338-958a-ebabf2c137ba" containerName="registry" probeResult="failure" output="Get \"https://10.134.0.13:5000/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.019021    4456 patch_prober.go:29] interesting pod/image-registry-59767d8795-7vwkt container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.134.0.13:5000/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.019064    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" podUID="0db36d92-4712-4338-958a-ebabf2c137ba" containerName="registry" probeResult="failure" output="Get \"https://10.134.0.13:5000/healthz\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.025170    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.031948    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-mxsjq" podUID="367d8d85-a5e8-462d-b1d7-91a01e62b726" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.059930    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": context deadline exceeded"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.065913    4456 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-tdx82 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.065953    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" podUID="0f2089f4-8ec7-45ce-a2df-0bf024b4158f" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.080928    4456 patch_prober.go:29] interesting pod/dns-default-zmrdt container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.134.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.080964    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-zmrdt" podUID="f59eba5d-a500-462e-9ecd-4ef9e553f873" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.103933    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-mxsjq" podUID="367d8d85-a5e8-462d-b1d7-91a01e62b726" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.206801    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.371648    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.553102    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:02.776944    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": context deadline exceeded"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:04.171762    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:04.852038    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:04.852766    4456 patch_prober.go:29] interesting pod/dns-default-zmrdt container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:04.852870    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-zmrdt" podUID="f59eba5d-a500-462e-9ecd-4ef9e553f873" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:04.882017    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:04.929101    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:04.929146    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:05.055569    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:05.055622    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:06.150277    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" podUID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:06.165009    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:06.165271    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:06.168954    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:06.169011    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:06.524273    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="83deda5b-6ef6-4a66-a76c-888303e6e834" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:06.524483    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="83deda5b-6ef6-4a66-a76c-888303e6e834" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:06.524208    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-6bfcf" podUID="da9f1b66-0e1b-4c7e-a947-a897c00ccc19" containerName="extractor" probeResult="failure" output="command timed out"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:06.677740    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:06.822688    4456 generic.go:356] "Generic (PLEG): container finished" podID="6957a218-62fc-4dd8-892e-4477b08c046f" containerID="0cb78ab8ebb38230ce58498878fbee1c9524a897317015a5ff7bf09d770bfdec" exitCode=1
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:06.865730    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:07.655530    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:07.913965    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:09.945406    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:09.945535    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:10.848980    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:10.849077    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.017853    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": context deadline exceeded"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.224415    4456 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-tdx82 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.224514    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" podUID="0f2089f4-8ec7-45ce-a2df-0bf024b4158f" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.324852    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": dial tcp 10.134.0.28:8081: connect: connection refused" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.324899    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": dial tcp 10.134.0.28:8081: connect: connection refused"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.365673    4456 scope.go:122] "RemoveContainer" containerID="0cb78ab8ebb38230ce58498878fbee1c9524a897317015a5ff7bf09d770bfdec"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.526666    4456 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.526724    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:05:12.561524    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.561932    4456 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.561959    4456 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.561973    4456 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.561994    4456 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeReady"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.569958    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": dial tcp 10.134.0.20:8081: connect: connection refused"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.580305    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" podUID="44b66f41-99ae-446b-a50a-2ec0b71563a4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.189:50052\" within 1s: context deadline exceeded"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:05:12.580423    4456 logging.go:55] [core] [Channel #6063 SubChannel #6064]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.189:50052", ServerName: "10.134.0.189:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.189:50052: operation was canceled"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.586145    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:05:12.606271    4456 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="10.738s"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.606375    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.606453    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.653127    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-hr8vz" podUID="e73be583-2a26-4379-8169-0a6b8784a11b" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.134.0.26:9403/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.658379    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-hr8vz" podUID="e73be583-2a26-4379-8169-0a6b8784a11b" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.134.0.26:9403/livez\": dial tcp 10.134.0.26:9403: connect: connection refused"
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.663037    4456 generic.go:356] "Generic (PLEG): container finished" podID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerID="98bb1cdd787c025aa16ef0cb488e4b052f1a1c499a91687b7d53b720295e0af2" exitCode=1
Nov 02 18:05:13.433065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.687767    4456 generic.go:356] "Generic (PLEG): container finished" podID="e73be583-2a26-4379-8169-0a6b8784a11b" containerID="f102af8d8427f21ccd5327dba082268df8744b692aa3dbb2752b19099048e51c" exitCode=1
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.694069    4456 scope.go:122] "RemoveContainer" containerID="98bb1cdd787c025aa16ef0cb488e4b052f1a1c499a91687b7d53b720295e0af2"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.694873    4456 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.696582    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="prometheus-operator-admission-webhook" containerStatusID={"Type":"cri-o","ID":"31ea127575345103ee79ddb9337e7680f26be2c47d1f8b1c1b65da3c280b1211"} pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" containerMessage="Container prometheus-operator-admission-webhook failed liveness probe, will be restarted"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.696642    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" containerID="cri-o://31ea127575345103ee79ddb9337e7680f26be2c47d1f8b1c1b65da3c280b1211" gracePeriod=30
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.717604    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerName="istio-proxy" containerID="cri-o://6398cfda523a8a8e2e673b041a862c3046065aa1b5573d8b8c734cf872ee2401" gracePeriod=30
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.730028    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.736533    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="metrics-server" containerStatusID={"Type":"cri-o","ID":"c426b7d4950263c913e37548ad7992f0212cc9969a4598d6b4fc3ff560c31e70"} pod="openshift-monitoring/metrics-server-7d987f75-ntp98" containerMessage="Container metrics-server failed liveness probe, will be restarted"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.736597    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/metrics-server-7d987f75-ntp98" podUID="18d2ae0a-bf65-4aee-8730-093e00b54a24" containerName="metrics-server" containerID="cri-o://c426b7d4950263c913e37548ad7992f0212cc9969a4598d6b4fc3ff560c31e70" gracePeriod=170
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.771458    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.771510    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" event={"ID":"6957a218-62fc-4dd8-892e-4477b08c046f","Type":"ContainerDied","Data":"0cb78ab8ebb38230ce58498878fbee1c9524a897317015a5ff7bf09d770bfdec"}
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.771540    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" event={"ID":"ec905c34-6dfb-483f-bb3d-7ba412262584","Type":"ContainerDied","Data":"98bb1cdd787c025aa16ef0cb488e4b052f1a1c499a91687b7d53b720295e0af2"}
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:05:12.771778    4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec905c34_6dfb_483f_bb3d_7ba412262584.slice/crio-98bb1cdd787c025aa16ef0cb488e4b052f1a1c499a91687b7d53b720295e0af2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec905c34_6dfb_483f_bb3d_7ba412262584.slice/crio-conmon-98bb1cdd787c025aa16ef0cb488e4b052f1a1c499a91687b7d53b720295e0af2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73be583_2a26_4379_8169_0a6b8784a11b.slice/crio-conmon-f102af8d8427f21ccd5327dba082268df8744b692aa3dbb2752b19099048e51c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ea3dac_7073_4564_b247_89ea94a64342.slice/crio-cdd773afd598fd21cda432d2c443c479f7282ad8cf4c3c694738ed4f0698ba92.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ea3dac_7073_4564_b247_89ea94a64342.slice/crio-conmon-cdd773afd598fd21cda432d2c443c479f7282ad8cf4c3c694738ed4f0698ba92.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73be583_2a26_4379_8169_0a6b8784a11b.slice/crio-f102af8d8427f21ccd5327dba082268df8744b692aa3dbb2752b19099048e51c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:05:12.772744    4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6957a218_62fc_4dd8_892e_4477b08c046f.slice/crio-conmon-0cb78ab8ebb38230ce58498878fbee1c9524a897317015a5ff7bf09d770bfdec.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec905c34_6dfb_483f_bb3d_7ba412262584.slice/crio-conmon-98bb1cdd787c025aa16ef0cb488e4b052f1a1c499a91687b7d53b720295e0af2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ea3dac_7073_4564_b247_89ea94a64342.slice/crio-cdd773afd598fd21cda432d2c443c479f7282ad8cf4c3c694738ed4f0698ba92.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73be583_2a26_4379_8169_0a6b8784a11b.slice/crio-f102af8d8427f21ccd5327dba082268df8744b692aa3dbb2752b19099048e51c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec905c34_6dfb_483f_bb3d_7ba412262584.slice/crio-98bb1cdd787c025aa16ef0cb488e4b052f1a1c499a91687b7d53b720295e0af2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6957a218_62fc_4dd8_892e_4477b08c046f.slice/crio-0cb78ab8ebb38230ce58498878fbee1c9524a897317015a5ff7bf09d770bfdec.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73be583_2a26_4379_8169_0a6b8784a11b.slice/crio-conmon-f102af8d8427f21ccd5327dba082268df8744b692aa3dbb2752b19099048e51c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ea3dac_7073_4564_b247_89ea94a64342.slice/crio-conmon-cdd773afd598fd21cda432d2c443c479f7282ad8cf4c3c694738ed4f0698ba92.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:05:12.775342    4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ea3dac_7073_4564_b247_89ea94a64342.slice/crio-cdd773afd598fd21cda432d2c443c479f7282ad8cf4c3c694738ed4f0698ba92.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6957a218_62fc_4dd8_892e_4477b08c046f.slice/crio-conmon-0cb78ab8ebb38230ce58498878fbee1c9524a897317015a5ff7bf09d770bfdec.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6957a218_62fc_4dd8_892e_4477b08c046f.slice/crio-0cb78ab8ebb38230ce58498878fbee1c9524a897317015a5ff7bf09d770bfdec.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec905c34_6dfb_483f_bb3d_7ba412262584.slice/crio-98bb1cdd787c025aa16ef0cb488e4b052f1a1c499a91687b7d53b720295e0af2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec905c34_6dfb_483f_bb3d_7ba412262584.slice/crio-conmon-98bb1cdd787c025aa16ef0cb488e4b052f1a1c499a91687b7d53b720295e0af2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73be583_2a26_4379_8169_0a6b8784a11b.slice/crio-f102af8d8427f21ccd5327dba082268df8744b692aa3dbb2752b19099048e51c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73be583_2a26_4379_8169_0a6b8784a11b.slice/crio-conmon-f102af8d8427f21ccd5327dba082268df8744b692aa3dbb2752b19099048e51c.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.776885    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.776934    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.776960    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hr8vz" event={"ID":"e73be583-2a26-4379-8169-0a6b8784a11b","Type":"ContainerDied","Data":"f102af8d8427f21ccd5327dba082268df8744b692aa3dbb2752b19099048e51c"}
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777015    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777037    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777057    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777102    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777138    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777175    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777211    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777254    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777291    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777330    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777374    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777419    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777463    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777486    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777525    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777584    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777632    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777681    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777726    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777773    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777858    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777908    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777956    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.777994    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.778032    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.778055    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.778097    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.778112    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:05:12.778907    4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec905c34_6dfb_483f_bb3d_7ba412262584.slice/crio-98bb1cdd787c025aa16ef0cb488e4b052f1a1c499a91687b7d53b720295e0af2.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73be583_2a26_4379_8169_0a6b8784a11b.slice/crio-conmon-f102af8d8427f21ccd5327dba082268df8744b692aa3dbb2752b19099048e51c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ea3dac_7073_4564_b247_89ea94a64342.slice/crio-conmon-cdd773afd598fd21cda432d2c443c479f7282ad8cf4c3c694738ed4f0698ba92.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6957a218_62fc_4dd8_892e_4477b08c046f.slice/crio-conmon-0cb78ab8ebb38230ce58498878fbee1c9524a897317015a5ff7bf09d770bfdec.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ea3dac_7073_4564_b247_89ea94a64342.slice/crio-cdd773afd598fd21cda432d2c443c479f7282ad8cf4c3c694738ed4f0698ba92.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73be583_2a26_4379_8169_0a6b8784a11b.slice/crio-f102af8d8427f21ccd5327dba082268df8744b692aa3dbb2752b19099048e51c.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec905c34_6dfb_483f_bb3d_7ba412262584.slice/crio-conmon-98bb1cdd787c025aa16ef0cb488e4b052f1a1c499a91687b7d53b720295e0af2.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:05:12.803030    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e1922175d5513058013bf3b85e2a3221900f1fde3d1509b5419561d2ef700c83\": container with ID starting with e1922175d5513058013bf3b85e2a3221900f1fde3d1509b5419561d2ef700c83 not found: ID does not exist" containerID="e1922175d5513058013bf3b85e2a3221900f1fde3d1509b5419561d2ef700c83"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:05:12.803082    4456 kubelet.go:1679] "Image garbage collection failed once. Stats initialization may not have completed yet" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"e1922175d5513058013bf3b85e2a3221900f1fde3d1509b5419561d2ef700c83\": container with ID starting with e1922175d5513058013bf3b85e2a3221900f1fde3d1509b5419561d2ef700c83 not found: ID does not exist"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.803143    4456 image_gc_manager.go:223] "Failed to monitor images" err="get container status: runtime container status: rpc error: code = NotFound desc = could not find container \"e1922175d5513058013bf3b85e2a3221900f1fde3d1509b5419561d2ef700c83\": container with ID starting with e1922175d5513058013bf3b85e2a3221900f1fde3d1509b5419561d2ef700c83 not found: ID does not exist"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.845110    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.846743    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"3041e8bd23c8eb7470f8dfba16bd766c148c2bcbf3b2edae11533a65a4f804ed"} pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" containerMessage="Container maas-api failed liveness probe, will be restarted"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.846968    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" containerID="cri-o://3041e8bd23c8eb7470f8dfba16bd766c148c2bcbf3b2edae11533a65a4f804ed" gracePeriod=30
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.858977    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": dial tcp 10.134.0.27:8081: connect: connection refused"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.873011    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": dial tcp 10.134.0.27:8081: connect: connection refused"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.891532    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.899706    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.899904    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.900062    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.900202    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.900306    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.900414    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.900515    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.900599    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.900693    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.900785    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.900937    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.901049    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.901136    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.901271    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zmrdt"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.901401    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.901506    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.901621    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.901778    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.902029    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.902197    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.902325    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.902502    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.906051    4456 scope.go:122] "RemoveContainer" containerID="f102af8d8427f21ccd5327dba082268df8744b692aa3dbb2752b19099048e51c"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.911984    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"d337c60ba714489232e4f5ede5270e793cd4779290b761e8f42568d05e170851"} pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" containerMessage="Container maas-api failed liveness probe, will be restarted"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.912043    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" containerID="cri-o://d337c60ba714489232e4f5ede5270e793cd4779290b761e8f42568d05e170851" gracePeriod=30
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.906396    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"cdd773afd598fd21cda432d2c443c479f7282ad8cf4c3c694738ed4f0698ba92"} pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" containerMessage="Container manager failed liveness probe, will be restarted"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.912238    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" containerID="cri-o://cdd773afd598fd21cda432d2c443c479f7282ad8cf4c3c694738ed4f0698ba92" gracePeriod=10
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.912365    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.912467    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.912546    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zmrdt"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.912589    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.916406    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.961522    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"e4e31f901eface112badee8c3415cf9f3881ca46d033fd3f068c3ec2f49f9ab9"} pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" containerMessage="Container maas-api failed liveness probe, will be restarted"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:12.961582    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" containerID="cri-o://e4e31f901eface112badee8c3415cf9f3881ca46d033fd3f068c3ec2f49f9ab9" gracePeriod=30
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.172962    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.198393    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.198761    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.199111    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.199215    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.199293    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": dial tcp 10.134.0.146:8443: connect: connection refused"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.199324    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": dial tcp 10.134.0.124:8443: connect: connection refused"
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.336036    4456 trace.go:236] Trace[676102884]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" (02-Nov-2026 18:04:38.912) (total time: 34422ms):
Nov 02 18:05:13.447751 ip-10-0-141-87 kubenswrapper[4456]: Trace[676102884]: [34.422995345s] [34.422995345s] END
Nov 02 18:05:13.747635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.733591    4456 generic.go:356] "Generic (PLEG): container finished" podID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerID="e4e31f901eface112badee8c3415cf9f3881ca46d033fd3f068c3ec2f49f9ab9" exitCode=0
Nov 02 18:05:13.747635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.733698    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" event={"ID":"6ebd3467-841a-4331-b149-8acf6e60c12c","Type":"ContainerDied","Data":"e4e31f901eface112badee8c3415cf9f3881ca46d033fd3f068c3ec2f49f9ab9"}
Nov 02 18:05:13.747635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.735915    4456 generic.go:356] "Generic (PLEG): container finished" podID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerID="d337c60ba714489232e4f5ede5270e793cd4779290b761e8f42568d05e170851" exitCode=0
Nov 02 18:05:13.747635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.736059    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" event={"ID":"9051e10f-854f-4cc8-90a7-dda319c49c9a","Type":"ContainerDied","Data":"d337c60ba714489232e4f5ede5270e793cd4779290b761e8f42568d05e170851"}
Nov 02 18:05:13.747635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.738285    4456 generic.go:356] "Generic (PLEG): container finished" podID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerID="3041e8bd23c8eb7470f8dfba16bd766c148c2bcbf3b2edae11533a65a4f804ed" exitCode=0
Nov 02 18:05:13.747635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.738466    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" event={"ID":"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f","Type":"ContainerDied","Data":"3041e8bd23c8eb7470f8dfba16bd766c148c2bcbf3b2edae11533a65a4f804ed"}
Nov 02 18:05:13.747635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.740732    4456 generic.go:356] "Generic (PLEG): container finished" podID="a1ea3dac-7073-4564-b247-89ea94a64342" containerID="cdd773afd598fd21cda432d2c443c479f7282ad8cf4c3c694738ed4f0698ba92" exitCode=0
Nov 02 18:05:13.747635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.740880    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" event={"ID":"a1ea3dac-7073-4564-b247-89ea94a64342","Type":"ContainerDied","Data":"cdd773afd598fd21cda432d2c443c479f7282ad8cf4c3c694738ed4f0698ba92"}
Nov 02 18:05:13.747635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.744865    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" event={"ID":"6957a218-62fc-4dd8-892e-4477b08c046f","Type":"ContainerStarted","Data":"50594c76d72639934489d5c1e00e5b464a34d6a50aa2d64165ad25bf4c978a07"}
Nov 02 18:05:13.747635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.747580    4456 generic.go:356] "Generic (PLEG): container finished" podID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerID="31ea127575345103ee79ddb9337e7680f26be2c47d1f8b1c1b65da3c280b1211" exitCode=0
Nov 02 18:05:13.759391 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.748808    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" event={"ID":"cae854c8-88f7-4201-b5d7-4e7c59fdf833","Type":"ContainerDied","Data":"31ea127575345103ee79ddb9337e7680f26be2c47d1f8b1c1b65da3c280b1211"}
Nov 02 18:05:13.759391 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:13.756419    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="cert-manager/cert-manager-597b95cb84-hr8vz"
Nov 02 18:05:14.038716 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.038677    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:14.105531 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.103014    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"]
Nov 02 18:05:14.234970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.233113    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"]
Nov 02 18:05:14.234970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.233612    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"
Nov 02 18:05:14.253471 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.253440    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:05:14.253620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.253600    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:05:14.363855 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.363507    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb80eb43-a6a5-4388-8436-6618c2f1493d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07748554-wnr4k\" (UID: \"bb80eb43-a6a5-4388-8436-6618c2f1493d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"
Nov 02 18:05:14.363855 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.363627    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb80eb43-a6a5-4388-8436-6618c2f1493d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07748554-wnr4k\" (UID: \"bb80eb43-a6a5-4388-8436-6618c2f1493d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"
Nov 02 18:05:14.363855 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.363683    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l98z9\" (UniqueName: \"kubernetes.io/projected/bb80eb43-a6a5-4388-8436-6618c2f1493d-kube-api-access-l98z9\") pod \"maas-api-revoke-sub-07748554-wnr4k\" (UID: \"bb80eb43-a6a5-4388-8436-6618c2f1493d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"
Nov 02 18:05:14.465880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.465828    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb80eb43-a6a5-4388-8436-6618c2f1493d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07748554-wnr4k\" (UID: \"bb80eb43-a6a5-4388-8436-6618c2f1493d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"
Nov 02 18:05:14.466077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.465946    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb80eb43-a6a5-4388-8436-6618c2f1493d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07748554-wnr4k\" (UID: \"bb80eb43-a6a5-4388-8436-6618c2f1493d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"
Nov 02 18:05:14.466077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.465996    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l98z9\" (UniqueName: \"kubernetes.io/projected/bb80eb43-a6a5-4388-8436-6618c2f1493d-kube-api-access-l98z9\") pod \"maas-api-revoke-sub-07748554-wnr4k\" (UID: \"bb80eb43-a6a5-4388-8436-6618c2f1493d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"
Nov 02 18:05:14.611508 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.611374    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb80eb43-a6a5-4388-8436-6618c2f1493d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-07748554-wnr4k\" (UID: \"bb80eb43-a6a5-4388-8436-6618c2f1493d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"
Nov 02 18:05:14.766337 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.766297    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" event={"ID":"a1ea3dac-7073-4564-b247-89ea94a64342","Type":"ContainerStarted","Data":"f46a45f18a610493bf46d8787ee563938a877ab2e179a2e516f44f6e70fcc573"}
Nov 02 18:05:14.766617 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.766596    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 18:05:14.770230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.769775    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" event={"ID":"cae854c8-88f7-4201-b5d7-4e7c59fdf833","Type":"ContainerStarted","Data":"5b3b2a55035fb65589abdc8d0eb062b82d79bbb8b490a8438c8c6505f3ee9c32"}
Nov 02 18:05:14.939986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.770364    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 18:05:14.939986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.770742    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": dial tcp 10.134.0.11:8443: connect: connection refused" start-of-body=
Nov 02 18:05:14.939986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.779864    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": dial tcp 10.134.0.11:8443: connect: connection refused"
Nov 02 18:05:14.939986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.783510    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hr8vz" event={"ID":"e73be583-2a26-4379-8169-0a6b8784a11b","Type":"ContainerStarted","Data":"d381788d1f97c945f12e6a0f9ca8aea3a3f0d92249a339bd95400d14c394ce40"}
Nov 02 18:05:14.939986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.783540    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 18:05:14.939986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:14.793723    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:15.552651 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:15.544113    4456 patch_prober.go:29] interesting pod/metrics-server-7d987f75-ntp98 container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok
Nov 02 18:05:15.552651 ip-10-0-141-87 kubenswrapper[4456]: [+]log ok
Nov 02 18:05:15.552651 ip-10-0-141-87 kubenswrapper[4456]: [+]poststarthook/max-in-flight-filter ok
Nov 02 18:05:15.552651 ip-10-0-141-87 kubenswrapper[4456]: [+]poststarthook/storage-object-count-tracker-hook ok
Nov 02 18:05:15.552651 ip-10-0-141-87 kubenswrapper[4456]: [-]metric-collection-timely failed: reason withheld
Nov 02 18:05:15.552651 ip-10-0-141-87 kubenswrapper[4456]: [+]metadata-informer-sync ok
Nov 02 18:05:15.552651 ip-10-0-141-87 kubenswrapper[4456]: livez check failed
Nov 02 18:05:15.552651 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:15.544198    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-7d987f75-ntp98" podUID="18d2ae0a-bf65-4aee-8730-093e00b54a24" containerName="metrics-server" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Nov 02 18:05:15.684191 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:15.682915    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb80eb43-a6a5-4388-8436-6618c2f1493d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-07748554-wnr4k\" (UID: \"bb80eb43-a6a5-4388-8436-6618c2f1493d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"
Nov 02 18:05:15.686431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:15.686399    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l98z9\" (UniqueName: \"kubernetes.io/projected/bb80eb43-a6a5-4388-8436-6618c2f1493d-kube-api-access-l98z9\") pod \"maas-api-revoke-sub-07748554-wnr4k\" (UID: \"bb80eb43-a6a5-4388-8436-6618c2f1493d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"
Nov 02 18:05:15.840081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:15.839936    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 18:05:15.958184 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:15.953897    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"
Nov 02 18:05:15.992836 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:15.988318    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:16.290112 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:05:16.290073    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:05:16.742360 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:16.742181    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:16.859416 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:16.859372    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" event={"ID":"9051e10f-854f-4cc8-90a7-dda319c49c9a","Type":"ContainerStarted","Data":"e4828365cff34649b08e97b25ad67554fb1835933f1008be8e43bed2f20e7560"}
Nov 02 18:05:16.862047 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:16.861216    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 18:05:16.866600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:16.865467    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" event={"ID":"ec905c34-6dfb-483f-bb3d-7ba412262584","Type":"ContainerStarted","Data":"a3e5b342fbd24d60a087ce776f7958683a916f3ab12b9d0f407b62ce53f0987a"}
Nov 02 18:05:16.866600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:16.866536    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 18:05:16.869978 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:16.869939    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" event={"ID":"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f","Type":"ContainerStarted","Data":"b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436"}
Nov 02 18:05:16.870599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:16.870572    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 18:05:16.874565 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:16.874040    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" event={"ID":"6ebd3467-841a-4331-b149-8acf6e60c12c","Type":"ContainerStarted","Data":"e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317"}
Nov 02 18:05:16.875704 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:16.875689    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 18:05:16.891440 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:16.891410    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:18.373887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.373804    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:18.386599 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:05:18.386555    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbb80eb43_a6a5_4388_8436_6618c2f1493d.slice/crio-29cb05c5086b14ad64a8e2a65f7359ddec707fd0f7ea68f38bd6ab1561e4d304 WatchSource:0}: Error finding container 29cb05c5086b14ad64a8e2a65f7359ddec707fd0f7ea68f38bd6ab1561e4d304: Status 404 returned error can't find the container with id 29cb05c5086b14ad64a8e2a65f7359ddec707fd0f7ea68f38bd6ab1561e4d304
Nov 02 18:05:18.386736 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.385888    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"]
Nov 02 18:05:18.518743 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.518636    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 18:05:18.626446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.594198    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-envoy\") pod \"a4323971-4e6e-43f6-bc29-49cbe5f12949\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") "
Nov 02 18:05:18.626446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.594313    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-credential-socket\") pod \"a4323971-4e6e-43f6-bc29-49cbe5f12949\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") "
Nov 02 18:05:18.626446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.594346    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-data\") pod \"a4323971-4e6e-43f6-bc29-49cbe5f12949\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") "
Nov 02 18:05:18.626446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.594396    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-workload-certs\") pod \"a4323971-4e6e-43f6-bc29-49cbe5f12949\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") "
Nov 02 18:05:18.626446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.594498    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-podinfo\") pod \"a4323971-4e6e-43f6-bc29-49cbe5f12949\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") "
Nov 02 18:05:18.626446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.594556    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a4323971-4e6e-43f6-bc29-49cbe5f12949-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4323971-4e6e-43f6-bc29-49cbe5f12949-istiod-ca-cert\") pod \"a4323971-4e6e-43f6-bc29-49cbe5f12949\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") "
Nov 02 18:05:18.626446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.594767    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-token\" (UniqueName: \"kubernetes.io/projected/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-token\") pod \"a4323971-4e6e-43f6-bc29-49cbe5f12949\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") "
Nov 02 18:05:18.626446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.594795    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a4323971-4e6e-43f6-bc29-49cbe5f12949-kube-api-access-26s6h\" (UniqueName: \"kubernetes.io/projected/a4323971-4e6e-43f6-bc29-49cbe5f12949-kube-api-access-26s6h\") pod \"a4323971-4e6e-43f6-bc29-49cbe5f12949\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") "
Nov 02 18:05:18.626446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.594926    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-workload-socket\") pod \"a4323971-4e6e-43f6-bc29-49cbe5f12949\" (UID: \"a4323971-4e6e-43f6-bc29-49cbe5f12949\") "
Nov 02 18:05:18.626446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.600751    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-workload-certs" pod "a4323971-4e6e-43f6-bc29-49cbe5f12949" (UID: "a4323971-4e6e-43f6-bc29-49cbe5f12949"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:05:18.626446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.625012    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-credential-socket" pod "a4323971-4e6e-43f6-bc29-49cbe5f12949" (UID: "a4323971-4e6e-43f6-bc29-49cbe5f12949"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:05:18.626446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.626323    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:05:18.626446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.626349    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:05:18.632727 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.631527    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a4323971-4e6e-43f6-bc29-49cbe5f12949-istiod-ca-cert" pod "a4323971-4e6e-43f6-bc29-49cbe5f12949" (UID: "a4323971-4e6e-43f6-bc29-49cbe5f12949"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:05:18.633385 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.633109    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-data" pod "a4323971-4e6e-43f6-bc29-49cbe5f12949" (UID: "a4323971-4e6e-43f6-bc29-49cbe5f12949"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:05:18.663992 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.663910    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-workload-socket" pod "a4323971-4e6e-43f6-bc29-49cbe5f12949" (UID: "a4323971-4e6e-43f6-bc29-49cbe5f12949"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:05:18.667113 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.666884    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-podinfo" pod "a4323971-4e6e-43f6-bc29-49cbe5f12949" (UID: "a4323971-4e6e-43f6-bc29-49cbe5f12949"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:05:18.674850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.671014    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-envoy" pod "a4323971-4e6e-43f6-bc29-49cbe5f12949" (UID: "a4323971-4e6e-43f6-bc29-49cbe5f12949"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:05:18.675914 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.675769    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4323971-4e6e-43f6-bc29-49cbe5f12949-kube-api-access-26s6h" pod "a4323971-4e6e-43f6-bc29-49cbe5f12949" (UID: "a4323971-4e6e-43f6-bc29-49cbe5f12949"). InnerVolumeSpecName "kube-api-access-26s6h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:05:18.683007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.682970    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-token" pod "a4323971-4e6e-43f6-bc29-49cbe5f12949" (UID: "a4323971-4e6e-43f6-bc29-49cbe5f12949"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:05:18.731069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.730173    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:05:18.731069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.730210    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:05:18.731069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.730224    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/a4323971-4e6e-43f6-bc29-49cbe5f12949-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:05:18.731069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.730236    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:05:18.731069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.730248    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-26s6h\" (UniqueName: \"kubernetes.io/projected/a4323971-4e6e-43f6-bc29-49cbe5f12949-kube-api-access-26s6h\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:05:18.731069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.730270    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:05:18.731069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.730283    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/a4323971-4e6e-43f6-bc29-49cbe5f12949-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:05:18.754274 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.754230    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:18.899830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:18.899729    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:19.001461 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.001425    4456 generic.go:356] "Generic (PLEG): container finished" podID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerID="6398cfda523a8a8e2e673b041a862c3046065aa1b5573d8b8c734cf872ee2401" exitCode=0
Nov 02 18:05:19.001951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.001923    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" event={"ID":"a4323971-4e6e-43f6-bc29-49cbe5f12949","Type":"ContainerDied","Data":"6398cfda523a8a8e2e673b041a862c3046065aa1b5573d8b8c734cf872ee2401"}
Nov 02 18:05:19.002573 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.002136    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b" event={"ID":"a4323971-4e6e-43f6-bc29-49cbe5f12949","Type":"ContainerDied","Data":"8b5a1359dbfc4bb66e3c6efb38ae746ffd5da85d3ce23a5e316fa1b4ccb878ce"}
Nov 02 18:05:19.002573 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.002178    4456 scope.go:122] "RemoveContainer" containerID="6398cfda523a8a8e2e673b041a862c3046065aa1b5573d8b8c734cf872ee2401"
Nov 02 18:05:19.002573 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.002380    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"
Nov 02 18:05:19.013278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.012695    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k" event={"ID":"bb80eb43-a6a5-4388-8436-6618c2f1493d","Type":"ContainerStarted","Data":"29cb05c5086b14ad64a8e2a65f7359ddec707fd0f7ea68f38bd6ab1561e4d304"}
Nov 02 18:05:19.037660 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.037631    4456 scope.go:122] "RemoveContainer" containerID="6398cfda523a8a8e2e673b041a862c3046065aa1b5573d8b8c734cf872ee2401"
Nov 02 18:05:19.038104 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:05:19.038079    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6398cfda523a8a8e2e673b041a862c3046065aa1b5573d8b8c734cf872ee2401\": container with ID starting with 6398cfda523a8a8e2e673b041a862c3046065aa1b5573d8b8c734cf872ee2401 not found: ID does not exist" containerID="6398cfda523a8a8e2e673b041a862c3046065aa1b5573d8b8c734cf872ee2401"
Nov 02 18:05:19.038567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.038111    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6398cfda523a8a8e2e673b041a862c3046065aa1b5573d8b8c734cf872ee2401"} err="failed to get container status \"6398cfda523a8a8e2e673b041a862c3046065aa1b5573d8b8c734cf872ee2401\": rpc error: code = NotFound desc = could not find container \"6398cfda523a8a8e2e673b041a862c3046065aa1b5573d8b8c734cf872ee2401\": container with ID starting with 6398cfda523a8a8e2e673b041a862c3046065aa1b5573d8b8c734cf872ee2401 not found: ID does not exist"
Nov 02 18:05:19.126606 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.126572    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:19.287170 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.287132    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:19.453930 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.453894    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:19.628327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.628226    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"]
Nov 02 18:05:19.659651 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.659615    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerName="istio-proxy"
Nov 02 18:05:19.659912 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.659895    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerName="istio-proxy"
Nov 02 18:05:19.685843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.660349    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" containerName="istio-proxy"
Nov 02 18:05:19.773901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.764596    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"
Nov 02 18:05:19.773901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.764496    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"]
Nov 02 18:05:19.773901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.770182    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w6-7df526-openshift-default-6d9cbcfc9c-fz74b"]
Nov 02 18:05:19.773901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.770213    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"]
Nov 02 18:05:19.781068 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.780858    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-227939\""
Nov 02 18:05:19.781068 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.781032    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-227939-dockercfg-4rtbn\""
Nov 02 18:05:19.831618 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.831581    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2\" (UID: \"97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"
Nov 02 18:05:19.831892 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.831715    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-sm4wt\" (UniqueName: \"kubernetes.io/projected/97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922-kube-api-access-sm4wt\") pod \"payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2\" (UID: \"97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"
Nov 02 18:05:19.934892 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.934850    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2\" (UID: \"97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"
Nov 02 18:05:19.935058 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.935014    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-sm4wt\" (UniqueName: \"kubernetes.io/projected/97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922-kube-api-access-sm4wt\") pod \"payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2\" (UID: \"97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"
Nov 02 18:05:19.937860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.937805    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922-plugins-config-volume\") pod \"payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2\" (UID: \"97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"
Nov 02 18:05:19.969402 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:19.969369    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-sm4wt\" (UniqueName: \"kubernetes.io/projected/97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922-kube-api-access-sm4wt\") pod \"payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2\" (UID: \"97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922\") " pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"
Nov 02 18:05:20.023214 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:20.023175    4456 generic.go:356] "Generic (PLEG): container finished" podID="bb80eb43-a6a5-4388-8436-6618c2f1493d" containerID="ddbb7b95206d8fc7e518db1afa3373123bf45a884a122a3a8a12139090ad5df4" exitCode=0
Nov 02 18:05:20.023405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:20.023251    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k" event={"ID":"bb80eb43-a6a5-4388-8436-6618c2f1493d","Type":"ContainerDied","Data":"ddbb7b95206d8fc7e518db1afa3373123bf45a884a122a3a8a12139090ad5df4"}
Nov 02 18:05:20.139349 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:20.139254    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"
Nov 02 18:05:20.403778 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:20.387572    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:20.484301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:20.484270    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"]
Nov 02 18:05:21.046258 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:21.046226    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" event={"ID":"97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922","Type":"ContainerStarted","Data":"7178d694cd88c8483d23321e761c3e272f6e8be0c2d9e09eb7e85fce91e1f001"}
Nov 02 18:05:21.046722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:21.046582    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" event={"ID":"97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922","Type":"ContainerStarted","Data":"551d09e335d3cc978253a0c35c85f21c1615ab274de47330246e1f404508b03b"}
Nov 02 18:05:21.046722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:21.046629    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"
Nov 02 18:05:21.097588 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:21.097532    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" podStartSLOduration=17.097516663 podStartE2EDuration="17.097516663s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:05:04 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:05:21.090777454 +0000 UTC m=+2137.581840616" watchObservedRunningTime="2026-11-02 18:05:21.097516663 +0000 UTC m=+2137.588579823"
Nov 02 18:05:21.334719 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:21.334600    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"
Nov 02 18:05:21.480687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:21.479216    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bb80eb43-a6a5-4388-8436-6618c2f1493d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb80eb43-a6a5-4388-8436-6618c2f1493d-maas-api-xks-ca\") pod \"bb80eb43-a6a5-4388-8436-6618c2f1493d\" (UID: \"bb80eb43-a6a5-4388-8436-6618c2f1493d\") "
Nov 02 18:05:21.480687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:21.479303    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bb80eb43-a6a5-4388-8436-6618c2f1493d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb80eb43-a6a5-4388-8436-6618c2f1493d-maas-api-openshift-ca\") pod \"bb80eb43-a6a5-4388-8436-6618c2f1493d\" (UID: \"bb80eb43-a6a5-4388-8436-6618c2f1493d\") "
Nov 02 18:05:21.480687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:21.479616    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bb80eb43-a6a5-4388-8436-6618c2f1493d-kube-api-access-l98z9\" (UniqueName: \"kubernetes.io/projected/bb80eb43-a6a5-4388-8436-6618c2f1493d-kube-api-access-l98z9\") pod \"bb80eb43-a6a5-4388-8436-6618c2f1493d\" (UID: \"bb80eb43-a6a5-4388-8436-6618c2f1493d\") "
Nov 02 18:05:21.486106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:21.486068    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bb80eb43-a6a5-4388-8436-6618c2f1493d-maas-api-openshift-ca" pod "bb80eb43-a6a5-4388-8436-6618c2f1493d" (UID: "bb80eb43-a6a5-4388-8436-6618c2f1493d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:05:21.488020 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:21.487986    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bb80eb43-a6a5-4388-8436-6618c2f1493d-maas-api-xks-ca" pod "bb80eb43-a6a5-4388-8436-6618c2f1493d" (UID: "bb80eb43-a6a5-4388-8436-6618c2f1493d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:05:21.488129 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:21.488035    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bb80eb43-a6a5-4388-8436-6618c2f1493d-kube-api-access-l98z9" pod "bb80eb43-a6a5-4388-8436-6618c2f1493d" (UID: "bb80eb43-a6a5-4388-8436-6618c2f1493d"). InnerVolumeSpecName "kube-api-access-l98z9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:05:21.583312 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:21.583277    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/bb80eb43-a6a5-4388-8436-6618c2f1493d-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:05:21.583312 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:21.583310    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l98z9\" (UniqueName: \"kubernetes.io/projected/bb80eb43-a6a5-4388-8436-6618c2f1493d-kube-api-access-l98z9\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:05:21.583525 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:21.583324    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/bb80eb43-a6a5-4388-8436-6618c2f1493d-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:05:21.619484 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:21.619414    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a4323971-4e6e-43f6-bc29-49cbe5f12949" path="/var/lib/kubelet/pods/a4323971-4e6e-43f6-bc29-49cbe5f12949/volumes"
Nov 02 18:05:22.058341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:22.058305    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"
Nov 02 18:05:22.058526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:22.058431    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k" event={"ID":"bb80eb43-a6a5-4388-8436-6618c2f1493d","Type":"ContainerDied","Data":"29cb05c5086b14ad64a8e2a65f7359ddec707fd0f7ea68f38bd6ab1561e4d304"}
Nov 02 18:05:22.058526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:22.058474    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="29cb05c5086b14ad64a8e2a65f7359ddec707fd0f7ea68f38bd6ab1561e4d304"
Nov 02 18:05:22.203010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:22.202972    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:05:23.037830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:23.037781    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 18:05:24.021703 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:24.021671    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 18:05:24.021910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:24.021752    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 18:05:24.507144 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:24.507105    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:06:09.967851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:06:09.672861    4456 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-pgmzg container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:06:53.801047 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:25.853875    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 18:06:53.801047 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:06:46.228827    4456 patch_prober.go:29] interesting pod/dns-default-zmrdt container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:06:53.801047 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:41.082107    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:08:18.720220 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:22.502896    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded"
Nov 02 18:08:18.720220 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:53.046944    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:08:18.720220 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:24.087739    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:08:18.720220 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:24.821579    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.43:50052: i/o timeout"
Nov 02 18:08:18.720220 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:25.153048    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": context deadline exceeded"
Nov 02 18:08:18.720220 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:25.366876    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.173:50052: i/o timeout"
Nov 02 18:08:18.720220 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:25.366935    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5" podUID="02a2dbe6-baeb-40c4-be0f-835239be331f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.135:50052: i/o timeout"
Nov 02 18:08:18.720220 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:25.367167    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" podUID="22b556c1-112b-49d4-932c-0143e912a54e" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.186:50052: i/o timeout"
Nov 02 18:08:18.720220 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:25.375892    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j" podUID="e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.44:50052: i/o timeout"
Nov 02 18:11:11.182668 ip-10-0-141-87 kubenswrapper[4456]: ift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" podUID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.172:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.182668 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:07:41.557703    4456 logging.go:55] [core] [Channel #6216 SubChannel #6278]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.172:50052", ServerName: "10.134.0.172:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.172:50052: operation was canceled"
Nov 02 18:11:11.182668 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.565202    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 02 18:11:11.182668 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.569526    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.182668 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.569639    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.182668 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.569936    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.183877 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.570263    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-863b87-openshift-default-dockercfg-5mwm4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.183877 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.570394    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rj9zp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.183877 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.570437    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.183877 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.570623    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.183877 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.570762    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fw7vt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.183877 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.570857    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xdl26\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.184546 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.570933    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.184546 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.570994    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.184546 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.571122    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"default-dockercfg-f7th6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.184546 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.571321    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mnv5k\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.184546 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.571504    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"default-dockercfg-h47km\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.184546 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.571596    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.185883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.571690    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qbqb7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.185883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.571798    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.185883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.571989    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"trusted-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.185883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.572113    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kqm5s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.185883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.572201    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dklz6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.185883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.572305    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"default-dockercfg-8x5zb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.186486 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.572433    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.186486 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.572512    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-557feb-dockercfg-nfpss\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.186486 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.572664    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-96768b-openshift-default-dockercfg-v8qql\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.186486 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.572763    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.186486 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.572950    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.186486 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.573077    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.187019 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.573190    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-j2fn5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.187019 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.573242    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.187019 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.573274    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.187019 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.573299    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-x9lxq\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.187019 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.573372    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mw4s9\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.187019 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.573442    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5mdnhf4neu495\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.187560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.573499    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.187560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.573562    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"default-dockercfg-q6667\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.187560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.573642    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.187560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.573712    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"cni-copy-resources\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.187560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.573775    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ppwdb\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.187560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.573778    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.188077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.573994    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.188077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.574154    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-96768b\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.188077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.574286    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mhkk4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.188077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.574467    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rdt6t\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.188077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.574746    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sn4p8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.188077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.574902    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.188627 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.574967    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vjnnw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.188627 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575027    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.188627 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575065    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.188627 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575110    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5oaelllrcvii0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.188627 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575152    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.188627 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575200    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8a209f-dockercfg-g8ztf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.189226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575237    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-8hdubeblopv5q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.189226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575284    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.189226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575327    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9lngf\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.189226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575369    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.189226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575408    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.189226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575449    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-2672ed-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.197353 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575501    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fbbbf8-dockercfg-29vsx\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.197353 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575540    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.197353 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575586    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.197353 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575638    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.197353 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575693    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.197353 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575730    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.197942 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575770    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.197942 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575831    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-2899ab\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.197942 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575893    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.197942 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575933    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.197942 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.575983    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"premium-w3-5eedab-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.197942 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.576031    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.204944 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.576066    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-504de7\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.204944 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.576221    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-2899ab-dockercfg-lwvws\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.204944 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.576443    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.204944 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.576517    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.204944 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.576529    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"cacerts\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.204944 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.576544    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-f0a3b8-dockercfg-h2cjr\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.205699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.576753    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-872354-dockercfg-hw7xl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.205699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.576976    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"original-pull-secret\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.205699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.576985    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8a209f\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.205699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.577134    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bb6d2e\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.205699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.577224    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.205699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.577232    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.206806 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.577421    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2c9b1d-dockercfg-tk2wk\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.206806 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.577625    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.206806 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.577641    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-227939\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.206806 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.577878    4456 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.RuntimeClass" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.206806 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.578088    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.206806 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.578116    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.578144    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jl8vw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.578248    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.578288    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.578312    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.578360    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bl7zd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.578371    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207511 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.578389    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207511 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.578411    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207511 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.578417    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207511 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.578524    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207511 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.578627    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-bfthl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207511 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.578693    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207870 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.578854    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207870 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.579048    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207870 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.579217    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a5c1da\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207870 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.579262    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-84fvd\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.207870 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.579446    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208048 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.579574    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lkf8q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208048 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.579590    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gml5d\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208048 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.579652    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208048 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.579802    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208048 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.580131    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-872354-openshift-default-dockercfg-rdxd7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208048 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.580202    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.580344    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d5b905-openshift-default-dockercfg-vtgjv\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.580405    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.580426    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.580474    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"downloads-dockercfg-gclvz\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.580571    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8a209f-openshift-default-dockercfg-whvdh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.580645    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208576 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.580697    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208576 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.580900    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"simulator-w1-120394-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208576 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.580904    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208576 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.580953    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208576 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.581159    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208576 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.581227    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208778 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.581289    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208778 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.581417    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208778 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.581446    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208778 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.581560    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-c810ca-openshift-default-dockercfg-s9n8v\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208778 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.581649    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a29999-dockercfg-r7mt7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.208778 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.581693    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d5b905-dockercfg-sfklt\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209023 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.581737    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209023 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.581910    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fbbbf8-openshift-default-dockercfg-b9pj2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209023 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582103    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p5nzm\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209023 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582118    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-b91583-dockercfg-vbwks\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209023 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582221    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209023 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582255    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582274    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"distinct-2-w1-120394-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582303    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5fd3e0-openshift-default-dockercfg-kxlxh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582333    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5eedab-openshift-default-dockercfg-57c6m\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582356    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582372    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582424    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582480    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"premium-w1-120394-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582576    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582620    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"istiod-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582715    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582847    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a5c1da-openshift-default-dockercfg-gjtlc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.582964    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2c9b1d-openshift-default-dockercfg-v7bnw\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583032    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583138    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583218    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-504de7-dockercfg-d4zk4\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583241    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583268    4456 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.CSIDriver" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583323    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c4dbe3\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583376    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583527    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c4dbe3-dockercfg-qp9g8\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583556    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t2hrp\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583565    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zt2z5\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583598    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mwsx7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.209976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583712    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583803    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-557feb-openshift-default-dockercfg-lvqfg\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583833    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583846    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.583936    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584011    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-504de7-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584064    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d5b905\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210417 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584084    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5nstj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210417 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584112    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210417 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584249    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vqf9q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210417 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584270    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"distinct-w3-5eedab-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210417 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584353    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210417 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584479    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210617 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584555    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-96768b-dockercfg-dm54d\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210617 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584716    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210617 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584796    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210617 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584873    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210617 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584878    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qn5gh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210617 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584920    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2899ab-openshift-default-dockercfg-t6vss\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210919 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584942    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210919 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584958    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210919 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.584987    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210919 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585001    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210919 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585037    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.210919 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585066    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"telemeter-client\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585080    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"distinct-2-w5-863b87-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585119    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a5c1da-dockercfg-vlnwc\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585158    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585163    4456 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Node" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.580201    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585208    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"default-dockercfg-8dv9q\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585242    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a29999-openshift-default-dockercfg-6b7bh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585269    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qpmjh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585275    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585301    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585311    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585342    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585378    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-f0a3b8-serving-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585419    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-872354\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585430    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn2p2\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585490    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585519    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b2fc9f-openshift-default-dockercfg-tqz7r\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585546    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585578    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585611    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-069fa5-openshift-default-dockercfg-ckxtj\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585646    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-multus\"/\"multus-daemon-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585674    4456 reflector.go:578] "Warning: watch ended with error" reflector="k8s.io/client-go/informers/factory.go:161" type="*v1.Service" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585688    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a29999\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.211802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585124    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-120394-openshift-default-dockercfg-qq496\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212025 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585714    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212025 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585755    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"premium-w0-069fa5-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212025 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585760    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4b9fl\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212025 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585207    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212025 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585801    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2c9b1d\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212195 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585836    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-f59572-dockercfg-s5jw7\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212195 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585856    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-f0a3b8\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212195 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585873    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212195 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585909    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212195 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585344    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585925    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-557feb\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585966    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p2jm6\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585385    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.586022    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.212420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.586080    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-227939-dockercfg-4rtbn\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.585967    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-f59572\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.586150    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fbbbf8\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.586194    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.586233    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.586266    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-b91583\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212789 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.586303    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-nv7lh\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212789 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.586377    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212789 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.586427    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212789 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.586486    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"simulator-w3-5eedab-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.212789 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.586523    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.213258 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.586566    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"simulator-w5-863b87-kserve-self-signed-certs\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.213258 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.586610    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\"" type="*v1.ConfigMap" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.213258 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.586648    4456 reflector.go:578] "Warning: watch ended with error" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4679w\"" type="*v1.Secret" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.213258 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:07:41.590516    4456 logging.go:55] [core] [Channel #6209 SubChannel #6211]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.187:50052", ServerName: "10.134.0.187:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.187:50052: i/o timeout"
Nov 02 18:11:11.213258 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.590617    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.187:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.213481 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.592041    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.213481 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.598003    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.213481 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.598067    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.213481 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.598261    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.213481 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.598318    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.213681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.598925    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.201:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.213681 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:07:41.599125    4456 logging.go:55] [core] [Channel #6287 SubChannel #6288]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.201:50052", ServerName: "10.134.0.201:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.201:50052: operation was canceled"
Nov 02 18:11:11.213681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.599523    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.213681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.599558    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.213681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.603876    4456 reflector.go:578] "Warning: watch ended with error" reflector="pkg/kubelet/config/apiserver.go:66" type="*v1.Pod" err="an error on the server (\"unable to decode an event from the watch stream: http2: client connection lost\") has prevented the request from succeeding"
Nov 02 18:11:11.213897 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:06:09.859889    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": context deadline exceeded" start-of-body=
Nov 02 18:11:11.213897 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:38.479357    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.213897 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:24.741801    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" podUID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.183:50052: i/o timeout"
Nov 02 18:11:11.213897 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:25.375880    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.213897 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.758542    4456 trace.go:236] Trace[441921031]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" (02-Nov-2026 18:06:09.105) (total time: 92652ms):
Nov 02 18:11:11.213897 ip-10-0-141-87 kubenswrapper[4456]: Trace[441921031]: [1m32.652588494s] [1m32.652588494s] END
Nov 02 18:11:11.213897 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.774771    4456 trace.go:236] Trace[199558328]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2" (02-Nov-2026 18:07:22.033) (total time: 19741ms):
Nov 02 18:11:11.213897 ip-10-0-141-87 kubenswrapper[4456]: Trace[199558328]: [19.741089525s] [19.741089525s] END
Nov 02 18:11:11.214191 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.779629    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.214191 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.780955    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.214191 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.781006    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.214191 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.783346    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.214191 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.783389    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.214390 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.786321    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.201:50052: i/o timeout"
Nov 02 18:11:11.214390 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.786607    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.214390 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.786887    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.214390 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.787446    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.214390 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.788075    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.214673 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.788263    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" podUID="0db36d92-4712-4338-958a-ebabf2c137ba" containerName="registry" probeResult="failure" output="Get \"https://10.134.0.13:5000/healthz\": context deadline exceeded"
Nov 02 18:11:11.214673 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.788715    4456 trace.go:236] Trace[733473499]: "Calculate volume metrics of kube-api-access-kjmhh for pod openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2" (02-Nov-2026 18:06:11.691) (total time: 73840ms):
Nov 02 18:11:11.214673 ip-10-0-141-87 kubenswrapper[4456]: Trace[733473499]: [1m13.840275548s] [1m13.840275548s] END
Nov 02 18:11:11.214673 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.788744    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" podUID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.232:50052: i/o timeout"
Nov 02 18:11:11.214673 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.789197    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" podUID="0db36d92-4712-4338-958a-ebabf2c137ba" containerName="registry" probeResult="failure" output="Get \"https://10.134.0.13:5000/healthz\": context deadline exceeded"
Nov 02 18:11:11.214673 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.790375    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.214673 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.790411    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.215059 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.792404    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.134:50052: i/o timeout"
Nov 02 18:11:11.215059 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.797372    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.215059 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.797428    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.215059 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:25.003757    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.215059 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:45.605846    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.215260 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.801021    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.215260 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.801943    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.215260 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.802004    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.215260 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.802268    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.215260 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.802612    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.215446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.608571    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" podUID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.182:50052: i/o timeout"
Nov 02 18:11:11.215446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.808301    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.215446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:25.375939    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.176:50052: i/o timeout"
Nov 02 18:11:11.215446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.809690    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.197377962s: [/var/lib/containers/storage/overlay/c7562bd346d831342663c585fc20338a2b2e898c53786e7fb087a0324f4f7e13/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.215446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.812693    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": context deadline exceeded"
Nov 02 18:11:11.215446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.813180    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.567158266s: [/var/lib/containers/storage/overlay/84e2978c7bf6f8a82e9cb045dcd37b94011845204a16c93bb3912ff1c3d5582e/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.215687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.813874    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.215687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:23.863662    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.215687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.815017    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m32.75502797s: [/var/lib/containers/storage/overlay/0234c330ad731526e0f7c2e86fd1692f4cb5053f776e389d790b7710b7d2e028/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.215687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.815073    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-mxsjq" podUID="367d8d85-a5e8-462d-b1d7-91a01e62b726" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.215687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.815764    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m33.569673871s: [/var/lib/containers/storage/overlay/ced9b4ee10b95de9e21ed7fe4450d4c9fcc97bb9e1d3ebda22fd9d469e5f4292/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.215687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.815917    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.010949468s: [/var/lib/containers/storage/overlay/9dcf2fbc8f2a12b867ec625655279863edbfabf7bb03acffd8482f4d3a0eae26/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.216150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.816001    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.011034048s: [/var/lib/containers/storage/overlay/1b20b5f4508512ef1b3778f562c5f97da26da6bf4d0856c4ef9a07f44819b3f0/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.216150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.816049    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.011081929s: [/var/lib/containers/storage/overlay/4756e8c484bb99f308a787d9771c513734c834c31e16597655b029fb5dc0d243/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.216150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.816087    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.01111774s: [/var/lib/containers/storage/overlay/7f098c561ddc879a9311701be103276e1d03f82ab3efa34f2b873a3e424f0d21/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.216150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.816163    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.011192084s: [/var/lib/containers/storage/overlay/da83a0515dd606978929d985eda34e4783473196836c68c43bf562fe9466bd7b/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.216150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.816205    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.011233586s: [/var/lib/containers/storage/overlay/4aff5b253dee34e4a614e7a9c0214a5c12d6acd592dd21d04cb698f2042a937c/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.216150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.816241    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.011269166s: [/var/lib/containers/storage/overlay/6304d0fceeb709e85ca572090857d08706f669cb1c3222985b10ace2393d51cf/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.216378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.816478    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m32.75651294s: [/var/lib/containers/storage/overlay/f41fd51d26fbd21de386cefe58613bae1900970a633ac15435e1675fb963123a/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.216378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.818938    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m32.758626082s: [/var/lib/containers/storage/overlay/1a394e1eaa16f81c47d4d9f50cab7c43ca075ce2f22fa2906b7916363672c95c/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.216378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.818985    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.013984601s: [/var/lib/containers/storage/overlay/b01ca7c42e592661866b33c6e00f21a35920d215fcb234779a7e876cf886a10f/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:11:11.216378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.819021    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m26.510603564s: [/var/lib/containers/storage/overlay/a5abae015c3465904cc5c7174a5b19880c528412ca15a33a710b819431e1c2f7/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.216378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.819114    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.859488822s: [/var/lib/containers/storage/overlay/4e67e8cd7fb897bc5116e504bba9a29a8ed30c4f9f1fac62d47ec5892d590847/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.216378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.819142    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" podUID="44b66f41-99ae-446b-a50a-2ec0b71563a4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.189:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.216603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.819449    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" podUID="22b556c1-112b-49d4-932c-0143e912a54e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.186:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.216603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.819609    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.859410741s: [/var/lib/containers/storage/overlay/2e0adf259b4f46a31705cecfb5e7924b04dcec1097b7bec6afcbc60c92bfa8a4/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.216603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:53.097954    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.216603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.821972    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.216603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.822137    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m31.076955245s: [/var/lib/containers/storage/overlay/0024ef56118e59543f04eba902323a6c84cea8671f97bb98441651fafd3f1fb9/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.216797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.822440    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.216797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.822537    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.859399507s: [/var/lib/containers/storage/overlay/cb2f4d21c7e74b313922bb09d6b5f3805778f08b0046d9b0e26544665af451da/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.216797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.823875    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.216797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.823910    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.216797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.827356    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.859430965s: [/var/lib/containers/storage/overlay/a18f6543a5970d2a5877e9cb019fa68742596c240618559891255c531214ebc1/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.216797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.827414    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.859392844s: [/var/lib/containers/storage/overlay/fab27cad89ad4b3e8d0b5c4ac0b7b3a7912aa4fa9886a06d4abd5720cb2eb720/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.217061 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.828640    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.859388778s: [/var/lib/containers/storage/overlay/76e05be6c6727d8ded946e030da1c274a89ca48d16a9caf69ad1842aaef2e487/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.217061 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.829199    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m1.778047742s: [/var/lib/containers/storage/overlay/8db47c776480e39a5f679a3dfbd86e84aed657e023ade55f978a8f68f01ea9f1/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.217061 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.829236    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m1.777925837s: [/var/lib/containers/storage/overlay/48451e9990851b8e024d8b3972a86ef0a0c9f6439dec07d6b3ef16f8fae2c355/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.217061 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.829266    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m1.777954749s: [/var/lib/containers/storage/overlay/08a91d46928182b4e183ee7ace17ce789177b554acc520da9618f6d1c993924e/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.217061 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.829463    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m1.778150173s: [/var/lib/containers/storage/overlay/2269cfd2edca6d6508ccccdbd523a9546a86e6f9de9c2f183ed3f2e8f50fb2fa/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.217061 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.829653    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m31.084470443s: [/var/lib/containers/storage/overlay/18864169905dab44c7b7c46db110c9b2e1d7a5b103538ab226d089b5c195ecec/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.217289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.829998    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.859392884s: [/var/lib/containers/storage/overlay/655a7e468823c9c970918a567d0b0ed61e5e4f40137739e6fa8ac19b456819a6/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.217289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.830151    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m30.859417352s: [/var/lib/containers/storage/overlay/c155031b458209f5c3cd2b2c6dbb1583f454b4f2ea510626f12b6fd5ab84995e/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.217289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.830203    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.217289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.830230    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.217289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.839808    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m31.446943502s: [/var/lib/containers/storage/overlay/a59d3ee737f274b70a661f619f4db78532081f4b90d92849748b1bc8db60f573/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.217289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.840158    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 58.672772512s: [/var/lib/containers/storage/overlay/64b4d87597c9acea5f8f6d9f7c7e5728a5c07b72e1d2ec2c99f1b9a6edfbacb0/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.218918 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.840233    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 58.672847732s: [/var/lib/containers/storage/overlay/fc1c8fde1913f2a15abbede07a2b0aad79a7cfd6834498affa92bcdb8a60d09b/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.218918 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.840291    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 58.672904198s: [/var/lib/containers/storage/overlay/9a23cb84ed43c601bc64e8f0c1b8d1f50c37c3e6f03c9342bea4efea4552db90/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.218918 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.842872    4456 trace.go:236] Trace[564437693]: "Calculate volume metrics of plugins-config-volume for pod openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" (02-Nov-2026 18:07:21.208) (total time: 20634ms):
Nov 02 18:11:11.218918 ip-10-0-141-87 kubenswrapper[4456]: Trace[564437693]: [20.634476617s] [20.634476617s] END
Nov 02 18:11:11.218918 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.844849    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-zmrdt" podUID="f59eba5d-a500-462e-9ecd-4ef9e553f873" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.218918 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:07:41.844953    4456 logging.go:55] [core] [Channel #6223 SubChannel #6294]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.186:50052", ServerName: "10.134.0.186:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.186:50052: operation was canceled"
Nov 02 18:11:11.218918 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.847792    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.219952 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.848226    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.200:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.219952 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:07:41.849558    4456 logging.go:55] [core] [Channel #6292 SubChannel #6293]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.189:50052", ServerName: "10.134.0.189:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.189:50052: operation was canceled"
Nov 02 18:11:11.219952 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:07:41.850941    4456 logging.go:55] [core] [Channel #6217 SubChannel #6301]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.200:50052", ServerName: "10.134.0.200:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.200:50052: operation was canceled"
Nov 02 18:11:11.219952 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.854392    4456 patch_prober.go:29] interesting pod/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.219952 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.854425    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.220456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:05:52.603412    4456 patch_prober.go:29] interesting pod/network-check-target-nttxj container/network-check-target-container namespace/openshift-network-diagnostics: Readiness probe status=failure output="Get \"http://10.134.0.5:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.220456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.855198    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d" containerName="network-check-target-container" probeResult="failure" output="Get \"http://10.134.0.5:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.220456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.606406    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.220456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.863209    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" podUID="e4c1132f-05b5-4664-84a2-ece57727b12a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.122:50052: i/o timeout"
Nov 02 18:11:11.220456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.865693    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" podUID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.183:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.220670 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:07:41.865892    4456 logging.go:55] [core] [Channel #6307 SubChannel #6310]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.183:50052", ServerName: "10.134.0.183:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.183:50052: operation was canceled"
Nov 02 18:11:11.220670 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.868173    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.220670 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.868944    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.170:50052: i/o timeout"
Nov 02 18:11:11.220670 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.868970    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" podUID="d837dde2-3939-4400-8ea7-fee58c162983" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.116:50052: i/o timeout"
Nov 02 18:11:11.220670 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.869788    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.175:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.221441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.875678    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.221441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.875704    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.42:50052: i/o timeout"
Nov 02 18:11:11.221441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.879268    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.129:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.221441 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:07:41.880025    4456 logging.go:55] [core] [Channel #6319 SubChannel #6320]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.129:50052", ServerName: "10.134.0.129:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.129:50052: operation was canceled"
Nov 02 18:11:11.221441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.880415    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" podUID="e4c1132f-05b5-4664-84a2-ece57727b12a" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.122:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.221666 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.881750    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.221666 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.881780    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.221666 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:07:41.882095    4456 logging.go:55] [core] [Channel #6324 SubChannel #6325]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.122:50052", ServerName: "10.134.0.122:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.122:50052: operation was canceled"
Nov 02 18:11:11.221666 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.882230    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.176:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.221666 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:07:41.882916    4456 logging.go:55] [core] [Channel #6329 SubChannel #6330]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.176:50052", ServerName: "10.134.0.176:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.176:50052: operation was canceled"
Nov 02 18:11:11.221904 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.883781    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.42:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.221904 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.885301    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded"
Nov 02 18:11:11.221904 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.886881    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": context deadline exceeded"
Nov 02 18:11:11.221904 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:07:41.910958    4456 logging.go:55] [core] [Channel #6309 SubChannel #6314]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.175:50052", ServerName: "10.134.0.175:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:60836->10.134.0.175:50052: use of closed network connection"
Nov 02 18:11:11.221904 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:07:41.924955    4456 logging.go:55] [core] [Channel #6333 SubChannel #6334]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.42:50052", ServerName: "10.134.0.42:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: write tcp 10.134.0.2:41022->10.134.0.42:50052: use of closed network connection
Nov 02 18:11:11.222113 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.981985    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 58.814566316s: [/var/lib/containers/storage/overlay/aff244db02eacd0334ea3e804c3284703be41cda08f02e8f8d0a2997784e5b04/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:11:11.222113 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.982041    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 58.814626653s: [/var/lib/containers/storage/overlay/70b4ec9c282135b6aad8758fa5f78d709532a56a100a349d35d742572e1db126/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:11:11.222113 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.982074    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 58.814661187s: [/var/lib/containers/storage/overlay/1c6d416b95253d64cbccfa7f2605c9d443f62eefa0f5d0626161a5261456fac0/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:11:11.222113 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.984894    4456 trace.go:236] Trace[1178177328]: "Calculate volume metrics of cni-binary-copy for pod openshift-multus/multus-additional-cni-plugins-ngkbz" (02-Nov-2026 18:07:25.543) (total time: 16441ms):
Nov 02 18:11:11.222113 ip-10-0-141-87 kubenswrapper[4456]: Trace[1178177328]: [16.441570712s] [16.441570712s] END
Nov 02 18:11:11.222113 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.985313    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.551569866s: [/var/lib/containers/storage/overlay/c7363f8222650c51f900a246744d12fb88176e629d384d106cefbae7c47dd7cd/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222113 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.985350    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.551602871s: [/var/lib/containers/storage/overlay/2a7f5aa5ba39a921bd515553a1089c5e8bb4a586c47eda778d4017ed30ecc198/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222113 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.985390    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.346457549s: [/var/lib/containers/storage/overlay/7049a5acdc2384d17d12da111ac7b7e2c0312fd1c17bd071788f8bf2808c768a/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.985419    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.34648524s: [/var/lib/containers/storage/overlay/95d0719995519e96ca4e2ef099d1cede945650fd1a2d8d4c03c7de64145ce3de/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.985443    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.346509157s: [/var/lib/containers/storage/overlay/e63da92cc01fe6e396c33175c254611578c02ed94dc2baa9164216cbb22bf034/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.985471    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.346535125s: [/var/lib/containers/storage/overlay/1c9abd01101f122b64b87bfa59945cc9d609da8965b554240c8857b6517c873e/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.985496    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.346558332s: [/var/lib/containers/storage/overlay/071d5a818923690326943183a456d5f42cc7cf06a45bbf46dce225490af0ccde/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:41.985528    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.346588106s: [/var/lib/containers/storage/overlay/8a34801be51a556958c8d17b00f8e1380149782e0320cbe6b256572a3ef0e18e/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.029603    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.082114424s: [/var/lib/containers/storage/overlay/959687ac2acb5f81ac7bb8a093e2668af6b0018b8cabcc37179678695dd639a3/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.029647    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.082159413s: [/var/lib/containers/storage/overlay/1c9f6f61c3280851db3a1425f487b201d7a593a4e27a560538129810b3169adc/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222938 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.029674    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.082184986s: [/var/lib/containers/storage/overlay/b428ce3434de02ee9be0b02f069e7c0e10cfc97ead5bb22bcc622bbc6836f7d3/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222938 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.029699    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.082208479s: [/var/lib/containers/storage/overlay/e64870204381c7a7f2555e3b5fe92951842ced33934d9f090c927aa4bb74212e/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222938 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.029723    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.082231578s: [/var/lib/containers/storage/overlay/a1ee19348961dae79c087e8ac3bf9e59ed7a33f37091eafee1f692eee68bc0a0/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222938 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.029752    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.08222977s: [/var/lib/containers/storage/overlay/ec89134dec345974daece69f9b8dd6137e843685bffddf149e15d295f298817e/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222938 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.029788    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.082264244s: [/var/lib/containers/storage/overlay/b85fed92aa7942c2d1b0a3ebd90904062437ad83478c28e6d785342de059e340/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222938 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.058545    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 47.700253621s: [/var/lib/containers/storage/overlay/8099f23161d5eec2e26bd9fd0d29e0723c872c3a2553491cef9562905ddcb497/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.222938 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.058589    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 47.700300123s: [/var/lib/containers/storage/overlay/25756dda2621604e75c6eea74f5fb75349668aa5a63ad77893fd3405a7c6f263/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.058641    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 47.700345142s: [/var/lib/containers/storage/overlay/c54941b7bb0f708eefe849545e48082649e74c106518b009fddf3e0544c6bc0f/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.058673    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 47.700381061s: [/var/lib/containers/storage/overlay/f43459bfc40568911db1c28d46def37ad81ebdd789dcbc7aef804efa5574b108/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.058706    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 47.700413594s: [/var/lib/containers/storage/overlay/7530ca161075d72465e087d6cfea7fe9079bd67c737753ce9107f7d5bf04700b/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.161325    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 22.896049852s: [/var/lib/containers/storage/overlay/fe9920e5a7065775346550e5358cc11abb4093be2682b270b828978d1fc3bdbb/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.195794    4456 patch_prober.go:29] interesting pod/metrics-server-7d987f75-ntp98 container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="HTTP probe failed with statuscode: 500" start-of-body=[+]ping ok
Nov 02 18:11:11.223205 ip-10-0-141-87 kubenswrapper[4456]: [+]log ok
Nov 02 18:11:11.223205 ip-10-0-141-87 kubenswrapper[4456]: [+]poststarthook/max-in-flight-filter ok
Nov 02 18:11:11.223205 ip-10-0-141-87 kubenswrapper[4456]: [+]poststarthook/storage-object-count-tracker-hook ok
Nov 02 18:11:11.223205 ip-10-0-141-87 kubenswrapper[4456]: [-]metric-collection-timely failed: reason withheld
Nov 02 18:11:11.223205 ip-10-0-141-87 kubenswrapper[4456]: [+]metadata-informer-sync ok
Nov 02 18:11:11.223205 ip-10-0-141-87 kubenswrapper[4456]: livez check failed
Nov 02 18:11:11.223205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.195861    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-7d987f75-ntp98" podUID="18d2ae0a-bf65-4aee-8730-093e00b54a24" containerName="metrics-server" probeResult="failure" output="HTTP probe failed with statuscode: 500"
Nov 02 18:11:11.223577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.319931    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.931281727s: [/var/lib/containers/storage/overlay/b193bcf543ea8163b7da56ebffe1a592dd1c275d976dd60b8d19c46f9f897138/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.319987    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.931342186s: [/var/lib/containers/storage/overlay/b4590eecb2fb63bb3d6ae41839b6301350c179ba2a20691909813f55406e0826/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.320021    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.931376251s: [/var/lib/containers/storage/overlay/4ab2f93d76b331d264037ae17bd9910369889be014d74845e96039454c48bbf7/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.320057    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.93141089s: [/var/lib/containers/storage/overlay/4835e8154ff3f70d907d90cedc5ad8524fba4d2106f30253d83cdcf05ba99c46/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.320088    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.931441533s: [/var/lib/containers/storage/overlay/d3bb1d8cdab5766b606d96bd1ebc61c634339df13a703533522356720f1907ac/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.320123    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.931475274s: [/var/lib/containers/storage/overlay/c3f17c66c7e230b3ba16f42aafdd369608de60e44df877a9cb942d9ed4c41247/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.320156    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.931506621s: [/var/lib/containers/storage/overlay/1f6300bdb8f1111be5619f5fe0108ffbf0081e9e27bd7de967b906b8b3ad33d1/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.320196    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.931544956s: [/var/lib/containers/storage/overlay/5517713a95fc08e0575503ff1195bf3084c018054f1947590ab93829560900bd/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.320229    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.931577433s: [/var/lib/containers/storage/overlay/76d53787e52a6a6617da848277a740036843744ce34ec997ac569eda8823e979/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.320263    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.931610553s: [/var/lib/containers/storage/overlay/4a89652208b5a1fd16b1a02605f67a9b20d35811800ea0052f9e3a8ea729ac7c/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.320296    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.931641296s: [/var/lib/containers/storage/overlay/39e7e1a3433145ba62e41a80782f5b1091cd38d1872bf5a708709525a34f7ff7/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.320325    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.931670475s: [/var/lib/containers/storage/overlay/78f0ff08138ee83d62f74d0f093a65703c9dba5248bba14dc94fce7c62d2eb68/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.320348    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.931692053s: [/var/lib/containers/storage/overlay/02ed3aed9d7d1f9932504189ec7d2043ab6ef5260de12ca49e1cf4bcedd555cf/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.223905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321194    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.931628631s: [/var/lib/containers/storage/overlay/dd180952a24151210f392620d65614521677a90358a32e8dcac6216703ef8c5e/diff ]; will not log again for this container unless duration exceeds 2s
Nov 02 18:11:11.224081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321227    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.922329889s: [/var/lib/containers/storage/overlay/f9027342e93f93cf2207e54c54a505f7bd2ec4fc18d0fb5e35c550a0d937439e/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321262    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.922361885s: [/var/lib/containers/storage/overlay/12a0c2eeda822e1ff661410a43076cdd6aedef1ccef4240d7456685f94696af9/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321300    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.922396866s: [/var/lib/containers/storage/overlay/47c6f9fa4e81d5b6bb6cb4211eff0df7ee33a694a7d6141f384651326b031a13/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321340    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.922437069s: [/var/lib/containers/storage/overlay/eea65d58e8e5145ae3222bd58cd763318191e7e1e8a3911be816c6f629e9177c/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321387    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.922482473s: [/var/lib/containers/storage/overlay/4e361aab9892c59ae1c857a4c3f26675ab711d42dbb2eea22c0038a24983687a/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321429    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.922523807s: [/var/lib/containers/storage/overlay/88bc22348df8e280411872f066dbf14feb3968b22f192d53027960ffae02080e/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321465    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.922558917s: [/var/lib/containers/storage/overlay/a7f26589ec75d499102c1e32470fe7d9c2e016680764619bc4f468b50424ee5f/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224296 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321630    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.922722849s: [/var/lib/containers/storage/overlay/7cd1c406d79080a5de52020433aba1bd1e3439731a6008c91a44ba0d734e0168/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224296 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321667    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.922758241s: [/var/lib/containers/storage/overlay/7a274702c3ffdc2535cd43c3814dd50ea0416d2fecf6b613833a80a6cd3be072/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224296 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321703    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.922793322s: [/var/lib/containers/storage/overlay/889a1e50cd346c2ae6c3699e22fc6333114a9fef99ca78329cab0059d0af21f8/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224296 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321739    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.922827701s: [/var/lib/containers/storage/overlay/12034bda1ff91295c47a5affae545ed5da549ac45cf44e53b8bd75905db1b189/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224296 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321775    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.922864071s: [/var/lib/containers/storage/overlay/e4f11f93cbfd2fc79389aec87c75cc993913f4c053e02fc2bef087f73afd5d7d/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224296 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321861    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 19.054707634s: [/var/lib/containers/storage/overlay/510d5b5021012ada853fd2e3b25638149887bac7a79939d0bf56c0262cc66081/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224296 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321908    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 17.499810569s: [/var/lib/containers/storage/overlay/f1bf059e369a60e8c8c66e453cf0724779ec1592faffe0ceeaccedee71095804/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321946    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.956078977s: [/var/lib/containers/storage/overlay/578d0af976732c1c9e43c84eae06aa0aa97b984f5931bc98ceceb74deb7f87b2/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.321984    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.852787621s: [/var/lib/containers/storage/overlay/8da893366900c89f1a49f0a89cff3a847547158c42b584792c9ed31f41e77cc2/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322021    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.852822836s: [/var/lib/containers/storage/overlay/bd48af5b0a05e1435415768044430abaf131384d551707641cd919d54da040a4/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322059    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.852858986s: [/var/lib/containers/storage/overlay/37d56634756aeeeca9171a043730fdb311d7d6352312d621fb8e69abd123ca4d/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322114    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.852912945s: [/var/lib/containers/storage/overlay/9f3bb7fa13dcbf6c99ec343710a35437e7f61d66d1bdac63538a05c396354291/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322153    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.852950199s: [/var/lib/containers/storage/overlay/8a61e7e207706ded21f782a468d9622ae8cfb244d32ad9c9d3f18363aca5ad52/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.224552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322187    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.852983227s: [/var/lib/containers/storage/overlay/229a9ce45461e83bcf894d5fdec3589a18c0dde28fd30e6c2f7267725d721843/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.225109 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322230    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.853015819s: [/var/lib/containers/storage/overlay/0a1b5af1d2fbdef3ff1b7294208f7f4115ccab6c1c6f56a4dbd94ef547fdbb15/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.225109 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322263    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.853055089s: [/var/lib/containers/storage/overlay/46c87d9588b5c4153befd46bcce313bb8d9eed33c45f7c4b76fbab7d7567da57/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.225109 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322320    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.853110335s: [/var/lib/containers/storage/overlay/b47b9dd6c514448c5e5a1ae04c109b9960c3981afb61d47419b7ec217051ba27/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.225109 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322356    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.853144892s: [/var/lib/containers/storage/overlay/9a65488a0d46fd5c20f43cfe794eb9f7883b34fba7149707cfc381b54e62c6dd/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.225109 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322372    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.853155815s: [/var/lib/containers/storage/overlay/2b04fc8b211dff465dfc3422cdf939d7911c23386635e5015c53667136caa0f0/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.225109 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322382    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 16.853168846s: [/var/lib/containers/storage/overlay/7e3710c7c30531a95b95d756958e640520b26b432243904fffd62b97f6e3862f/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.225109 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322403    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m38.221440359s: [/var/lib/containers/storage/overlay/e7f6af957917b0de1eac2c0f835eac9f4d66364bc548a8587261fc6bec5607da/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.225403 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322427    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m55.612051584s: [/var/lib/containers/storage/overlay/0d74f9f4fbbf600087eeb718bd1494b6cee4f94be2e06fed79e7cc0e32173cf4/diff /var/log/pods/openshift-dns_dns-default-zmrdt_f59eba5d-a500-462e-9ecd-4ef9e553f873/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.225403 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322448    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.690094556s: [/var/lib/containers/storage/overlay/1b551b04f397d2a17bb4b72d2a9e13ea7b255a0c2887bd5f17c4e17a3c5d3890/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/kube-rbac-proxy-node/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.225403 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322462    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.690105125s: [/var/lib/containers/storage/overlay/d0014aabc7632aacf8f230d1d606750084728ea64d88ba0a6c129361bdd715f8/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/thanos-sidecar/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.225403 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322476    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.690118072s: [/var/lib/containers/storage/overlay/39036cdcb7e7029ac2d15241aa50d868ff1a012fe62623885c01ed6f7528dbce/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.225403 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322491    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.690130635s: [/var/lib/containers/storage/overlay/afcf005d130d4b0994316011111d3ba89bede3db3572360e1b8cddd15c5a4284/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.225718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322510    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m26.226410229s: [/var/lib/containers/storage/overlay/6adcc40b109806b8d4dee9b3334b845a3feca9388fee349574a27397b371bd0e/diff /var/log/pods/openshift-jobset-operator_jobset-controller-manager-77cfb5f94-gfbr9_6957a218-62fc-4dd8-892e-4477b08c046f/manager/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:11:11.225718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322528    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.690206951s: [/var/lib/containers/storage/overlay/903a7a7c7002a9034e9b89db1ce800ddd9a85962e7522211d8fa957ab69f9ec0/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-m89vg_9257d59b-6aaf-423e-b7ed-2e64dbdfa535/kube-rbac-proxy-main/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.225718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322541    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.690201464s: [/var/lib/containers/storage/overlay/37c279f7138b64cf848e046dd0159412d2f60a6e3f561819a069d84c7835dd5e/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/ovnkube-controller/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.225718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322556    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.690204839s: [/var/lib/containers/storage/overlay/68adcccdea3eba205d654c160d713ccd9fe3f136335e3c8c4fd9800b1d4da9c1/diff /var/log/pods/openshift-network-diagnostics_network-check-target-nttxj_a974c7e3-bf97-4575-bec1-c09fcd018f0d/network-check-target-container/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226027 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322574    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.69022126s: [/var/lib/containers/storage/overlay/b7702443b08031bf70dfb72c9da1f832917de0a24b9b53011dbe509513501266/diff /var/log/pods/kube-system_global-pull-secret-syncer-mmhd7_8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d/global-pull-secret-syncer/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226027 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322604    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155217934s: [/var/lib/containers/storage/overlay/7c41892d8c349d1ad9ae2bbe72fa8e68344bac8db842e0614212577491c66d1d/diff /var/log/pods/openshift-monitoring_telemeter-client-79c994cf5c-ckj5h_e065a5c3-95fd-4ac8-a287-8e1d20219201/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226027 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322619    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155231408s: [/var/lib/containers/storage/overlay/da5b82203a9243942a4be5d58ab45616354c2ad53c09072b04c350a996fc6bd7/diff /var/log/pods/openshift-ingress_e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s_5bb882f6-fd8f-46ae-9396-b52498c5d9db/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226027 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322634    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.1552443s: [/var/lib/containers/storage/overlay/44d9303590300b2067d19221719afe781447e01e7a3698b47e26576b04692428/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/config-reloader/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226368 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322649    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155259449s: [/var/lib/containers/storage/overlay/436b55b989bc697feca6cac52a8bbeabf4607101e8e2d4cf93eca0342b9c313b/diff /var/log/pods/openshift-monitoring_node-exporter-r45c5_8c89d394-c878-4069-b3a9-cd0abc822fec/node-exporter/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226368 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322665    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.690310692s: [/var/lib/containers/storage/overlay/d5f8aa2b7ec7f078e5d3587bf3acd312cea2db5330296f8eeaf6747bb8ecb4af/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6_60c0e838-5c79-4bf7-a7c6-87903ae8bd8c/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226368 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322683    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.690326629s: [/var/lib/containers/storage/overlay/1d99eb2b6b4423a5208899af1d81194c5531b55c6df383327a20817e35f00610/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89_8c60dfaa-dabe-4726-91b5-165443ff3971/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226368 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322698    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155304819s: [/var/lib/containers/storage/overlay/349bb891311b428d7245edc3a45686b0ebfaa1beb5ddb02e1c5186f29d1a0a38/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/nbdb/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322711    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155318808s: [/var/lib/containers/storage/overlay/a568d4ed72e2a436fe819cab66f62bd95c9b47563cb98179484da40963eb831c/diff /var/log/pods/e2e-models-e2e-worker-w3-5eedab_premium-w3-5eedab-kserve-786db78999-f8lbq_f563607b-3be4-4194-aa10-81e8bf2cbf6e/main/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322727    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155336183s: [/var/lib/containers/storage/overlay/cf8db5461455b84412026ed2fb5a9f7f27755e4db156859b09da808cfbc028e3/diff /var/log/pods/e2e-models-e2e-worker-w5-863b87_distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f_bffc7364-3157-46e6-8fba-616f128b60cf/main/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322740    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155343361s: [/var/lib/containers/storage/overlay/30c6ef23d54157e2314449730cec7e09831a7cd2d10ec7e7cdc43a4ced04dd2a/diff /var/log/pods/openshift-monitoring_node-exporter-r45c5_8c89d394-c878-4069-b3a9-cd0abc822fec/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322758    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155360162s: [/var/lib/containers/storage/overlay/411d81d088de93b0674ea4f2dd71b25e85f37341baf36fa8546cc1221ec492ad/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322771    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155372755s: [/var/lib/containers/storage/overlay/4e57f3a2be8d4ab9b3c25af8d27d43230ec513dda8164ae6c781b961a859b92b/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/prometheus/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322784    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.517812266s: [/var/lib/containers/storage/overlay/0f64d44e017ec70c0f08927d7a5b5315efaaa27e26b52cb55a2d48db8ff58cb5/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4_18a32a7b-66fa-4f34-ab36-2beb0169e430/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322797    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155395037s: [/var/lib/containers/storage/overlay/a6efa294561d8867fca81b5aa12f5b9f4ce0955f3122e0d2c68924d7a6db53f8/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy-rules/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.226847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322809    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155414822s: [/var/lib/containers/storage/overlay/9f31774d55e55572596ad5b05f6824dca197d8d645a3a2091b071095f5fc348f/diff /var/log/pods/openshift-ingress_e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j_2dc48441-ccc0-47fc-abe7-7c327559eba4/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227091 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322845    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155440055s: [/var/lib/containers/storage/overlay/48d5c82df7e650a089ea0fa91ed0bd4c1b6a9b1f380d8d8db83882fc5cc1bf60/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-driver/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227091 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322865    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155459799s: [/var/lib/containers/storage/overlay/ec5326cd4c57e5289c85be1ed540648b29113e91e4b3f4a2026c4c66a8e87d65/diff /var/log/pods/openshift-ingress-canary_ingress-canary-n4r8t_ba5cbec9-7649-462a-b110-11ebcc5d6d78/serve-healthcheck-canary/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227091 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322884    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155480621s: [/var/lib/containers/storage/overlay/e7bd185adbe2a5f6c8ee6467194d0c245d24e52db000b36e4df509e6ff6b0e8a/diff /var/log/pods/openshift-ingress_e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq_778b9a36-6a7b-4251-a91a-999e358d8c5c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227091 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322899    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155491913s: [/var/lib/containers/storage/overlay/d002b0ce8a03e77141888fb5a4a50972cf6108b7e5e59c9375aa48cad9ced5af/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx_d837dde2-3939-4400-8ea7-fee58c162983/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322919    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m2.27160683s: [/var/lib/containers/storage/overlay/8df838514665e6e27e507b914dcdc2f57a13508b4557b623dfa7fb79412fd9b0/diff /var/log/pods/openshift-ingress_e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f_e5167f1d-3718-4c96-bf19-1cdc1ab73207/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322931    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.517958204s: [/var/lib/containers/storage/overlay/1a5df3100db9dd3cfd75b7f681887de0a6a1fe77ee9e7827781cfdc3b41a2d4d/diff /var/log/pods/openshift-ingress_e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck_694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322948    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155537405s: [/var/lib/containers/storage/overlay/7c82bb52ad040de109f8f96e4fde961296a9a03788caea2ee7088d4b4b9b113b/diff /var/log/pods/openshift-ingress_e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt_82ba53b4-e91a-4299-9498-f5abcd34576b/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322965    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m32.269557923s: [/var/lib/containers/storage/overlay/d15dc5bffc34599d4e17b91bff8ac1423d9077ae71856a7a2755b92a3bd6dcfc/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-5knvx_8ef79a10-292b-42d9-aab0-53bee3e43ed2/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227481 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322979    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155559025s: [/var/lib/containers/storage/overlay/833351d2ebd14fa986d0203e8af3254003326af193a6655f85b7084ad760a6ec/diff /var/log/pods/openshift-ingress_e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf_999df778-cd9b-4b41-852d-4661a7326483/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227481 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.322992    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155592346s: [/var/lib/containers/storage/overlay/42bab2822c3e5230896f9c86b84693de76f186c67813e61eb7792866accc56b3/diff /var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4_6e44698a-838e-4eea-b956-585673963ed9/main/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227481 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323009    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.684061459s: [/var/lib/containers/storage/overlay/ff286283e890b2f02d7b35cbc69ffb7a37f1ddbe5461c9aabad15b3d6f880cd6/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/kube-rbac-proxy-thanos/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227481 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323026    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.684077338s: [/var/lib/containers/storage/overlay/801517f9d9b867bed7e9f37f0f99b4b285f8fe58d0c6fe3b4e406c4b66eb885f/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g_c0ff0cbb-4acf-4d63-9579-6aeac659b2ea/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323049    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.690731015s: [/var/lib/containers/storage/overlay/0ba4937b318aff26f5d04de663e3cbc5fb91d34ecbaf5c1a70e275fa82d80f53/diff /var/log/pods/openshift-ingress_e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z_d4161075-440d-4373-b878-57c77612c6d2/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323062    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.68411191s: [/var/lib/containers/storage/overlay/87c11e77a69c1256c563023edeadf4b64a91805f1340898de6ce44397e54a787/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323076    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.684124274s: [/var/lib/containers/storage/overlay/932ab5e5702f5c7c79c2d32910577875654df1477eac0759bb03b85f11ce0d04/diff /var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-lxxf2_8064da0c-fc5d-493c-bcdf-4327716d4cc2/migrator/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:11:11.227681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323089    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155682525s: [/var/lib/containers/storage/overlay/968e208b4860faed3d3e26aa71e41adb5c447ce22ac3bb6cd22690346d2dfb5c/diff /var/log/pods/e2e-models-e2e-worker-w3-5eedab_distinct-w3-5eedab-kserve-856b8c7458-k2lgz_0874d0e8-5883-477d-b64a-d455934cfc90/main/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227891 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323104    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.69076721s: [/var/lib/containers/storage/overlay/45ab15387881b8da1f41ebe3c7b88bd293020499c9cacdb6a8c2b4a9b2ed2fb6/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg_b8847684-5676-4253-850f-9f2b2c42ea0c/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227891 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323153    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.690808505s: [/var/lib/containers/storage/overlay/acc66a71ede97842cda782e51e0adbea72945cd7fa0b1af2fc4160e227b7062f/diff /var/log/pods/openshift-ingress_e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5_0996b2fa-40e8-469c-9ace-75458691fa25/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227891 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323187    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.69086623s: [/var/lib/containers/storage/overlay/508796c136616a6a941f39c30320d167ca9f8cc36fab3ce4085e5e6164b8c836/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr_e141d845-db9d-4f94-b05e-cfb7848e8fef/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.227891 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323207    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.155810539s: [/var/lib/containers/storage/overlay/f62221cc1bb4ee783a4479f772f1b3eaee2ae9132e39581452d31149674b631a/diff /var/log/pods/e2e-models-e2e-worker-w3-5eedab_simulator-w3-5eedab-kserve-795c4575bb-f25x5_0c6cec58-b8d9-4856-9446-811407a747ab/main/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228188 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323236    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.690896892s: [/var/lib/containers/storage/overlay/67b48a8227291881bea04d0f599da8269338873aa7b51eb6d44f387b34a084de/diff /var/log/pods/e2e-models-e2e-worker-w5-863b87_simulator-w5-863b87-kserve-bd54699cd-7g2mv_76d3e197-83a0-4793-8593-0b47b55a9fc4/main/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228188 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323257    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m44.681642574s: [/var/lib/containers/storage/overlay/be0a87d2eccccae8e85688b90a61b88439e6fd706c76be4ba52b6f95c4beac73/diff /var/log/pods/openshift-ingress_e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz_408afe2d-75a0-4537-b58c-9ab31f161685/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228188 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323280    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058075506s: [/var/lib/containers/storage/overlay/f4bf622192145887ae6ee7c0202271dd59950c44754a850ccabfcb18cbe5c0a3/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-m89vg_9257d59b-6aaf-423e-b7ed-2e64dbdfa535/kube-rbac-proxy-self/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228188 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323299    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m44.78393072s: [/var/lib/containers/storage/overlay/79ff9c7f890e079eb19a8a8060e78ed7f0f7644efb29ded7f8d223b309ccdf2d/diff /var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k_14d43510-cfbb-4ddc-88cb-ccd4cb3a543d/main/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228430 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323319    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058103333s: [/var/lib/containers/storage/overlay/3c75f5f1fa0b98e778e718536ee383cfcc5ca413d1084810926fa3df6d132250/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5_02a2dbe6-baeb-40c4-be0f-835239be331f/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228430 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323338    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.691001972s: [/var/lib/containers/storage/overlay/07e8324700651cc1b533fd9bd9bd56d25f48721c5261994354ea99a5533a21f1/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx_38977edb-8612-48c6-a6c6-1714a50a9e4f/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228430 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323378    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058156413s: [/var/lib/containers/storage/overlay/6ea0c1ff6b4e328bbe49837fbf2858fecebf87243389fc2d2d8ab2b78934c93e/diff /var/log/pods/openshift-monitoring_telemeter-client-79c994cf5c-ckj5h_e065a5c3-95fd-4ac8-a287-8e1d20219201/telemeter-client/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228430 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323413    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058195904s: [/var/lib/containers/storage/overlay/54eb6dcdb0f244dbd1ab517bf5fd9b7b8ca4a049b4741da30b72d995007c9bd3/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g_44b66f41-99ae-446b-a50a-2ec0b71563a4/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228641 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323429    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058209162s: [/var/lib/containers/storage/overlay/d6bd7fb5658266e0883715f2708154a35f91862981569c388e3cb5f866afeeb3/diff /var/log/pods/openshift-ingress_e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7_f56cfa88-cc79-4d08-9111-b47d940533ba/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228641 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323463    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058241779s: [/var/lib/containers/storage/overlay/92e435ce4ca5afe43253221b15bf185141e8b9ccff9215e533a29abb68c8d01e/diff /var/log/pods/openshift-network-operator_iptables-alerter-dlp6w_6767bd29-3165-4e67-81b3-e320e8324034/iptables-alerter/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228641 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323483    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058259917s: [/var/lib/containers/storage/overlay/49fe87eab95fb5bcdb653a69186242a7bf84794358441a45b2ffc81148fc7f68/diff /var/log/pods/openshift-multus_network-metrics-daemon-tk8ht_dd733778-7002-4a26-a4e8-4525d8f6bee9/network-metrics-daemon/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228641 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323497    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m45.250533992s: [/var/lib/containers/storage/overlay/3e001268063ff1680111524029f98c7759f3a4feccf0afcf7064a2ca68c37a83/diff /var/log/pods/e2e-models-e2e-worker-w0-069fa5_premium-w0-069fa5-kserve-699ddf764c-xmspr_9e85cf3f-eb7a-452b-b716-26df5b391065/main/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228922 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323514    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058290319s: [/var/lib/containers/storage/overlay/9de66e4fccd5433ca81c75c6841035983055f1565dac16f5b02b99f158a7e535/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5_3d398f17-735a-4d4b-b1cc-5b24fd266535/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228922 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323537    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058310811s: [/var/lib/containers/storage/overlay/57432df54ab94c322fcc3f8ed7e5d69537ef205267b7e8198fece234918f9d44/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n_85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228922 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323555    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058327616s: [/var/lib/containers/storage/overlay/b8b56f59390f731005dd4f5e142899c74da144d464f61fbeea461b527f3dfc3d/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/prom-label-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.228922 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323569    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058340997s: [/var/lib/containers/storage/overlay/2baaae2b22b3f8e778df214acf8cc253c7a7bfa34a8e2a49f4c49e86ff018e9a/diff /var/log/pods/openshift-console_downloads-6dd789c8b7-pgmzg_6e78d6c7-c325-48f2-af90-804dade84d8f/download-server/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323583    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m44.605497071s: [/var/lib/containers/storage/overlay/8fe16763de809dbbc885c5aa483909cdeb3f142a735700e1976d0912af066878/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2_49238ac8-57fa-412a-88bd-681364a0b455/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323603    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058388664s: [/var/lib/containers/storage/overlay/9437d7121cf0584354b918af2df36d1e6733adeb8e3ad8e653ee51e2eaf2b268/diff /var/log/pods/e2e-models-e2e-worker-w1-120394_premium-w1-120394-kserve-669cf87f88-rmj4s_5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34/main/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323619    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m3.075137199s: [/var/lib/containers/storage/overlay/2451471e892c455bc0429337eed2450e847bf657dfb5b21286782c82632452ed/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz_b5d45c38-1cef-4aa9-a519-98aed00e8727/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323634    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058398267s: [/var/lib/containers/storage/overlay/d5d50d36d455b3124bba1a4682c6b556a344e4c0f72561f6ab14472bbe327aa9/diff /var/log/pods/openshift-multus_multus-additional-cni-plugins-ngkbz_c2d7ba52-9a2b-4871-aca0-3c4e1934c572/kube-multus-additional-cni-plugins/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323653    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058415305s: [/var/lib/containers/storage/overlay/613c01497ad5e2672a318b68ad56355e93f5e704f0918037c43a2d38d967a924/diff /var/log/pods/openshift-monitoring_telemeter-client-79c994cf5c-ckj5h_e065a5c3-95fd-4ac8-a287-8e1d20219201/reload/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323678    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058444803s: [/var/lib/containers/storage/overlay/35ea380649cc6d0f8dc0e907ecb9ef66961662f67283708985d685f75f06497f/diff /var/log/pods/openshift-ingress_e2e-worker-w1-120394-openshift-default-6459698f85-j24dq_acd44782-84e1-411a-9a1c-6d21456733bb/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323695    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.05845453s: [/var/lib/containers/storage/overlay/48a8a99b5aa38b5a1f8af93f2cf91f2afe257cea7b9b3fc9ad86c404aea4cfa8/diff /var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323709    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058467431s: [/var/lib/containers/storage/overlay/df02c5a738481c502ee0fbb85ca8fb2c47529fde2b1bfbb73efe51cb364c310b/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/ovn-controller/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323723    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.684770548s: [/var/lib/containers/storage/overlay/7840ca7106125a4b22fa575aa093373b243546e0cef5c452d1dd3a71532c03ea/diff /var/log/pods/openshift-ingress_e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt_856c2dca-bad2-4f62-a536-1ca0e09f0a12/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323748    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058512837s: [/var/lib/containers/storage/overlay/a55dca0d0429c95641f315985403a890eb29899befd46ff107f67affa1d6343d/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw_5b0ff069-2a17-489e-ab3d-6d5461bffbf4/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323766    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.684816247s: [/var/lib/containers/storage/overlay/05b2f9768ded3bb89850b28832c2c3463a2ffa7b9eab99ff39d4e91999d29770/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-tdx82_0f2089f4-8ec7-45ce-a2df-0bf024b4158f/discovery/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323779    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058560491s: [/var/lib/containers/storage/overlay/fdf2869abb29b990967a0fcf0bb0ef8a663dfc34c944a5841012b183fb830979/diff /var/log/pods/openshift-ingress_e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4_92ad0220-435f-4ae8-b743-3421ce98e6fb/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229613 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323825    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058560945s: [/var/lib/containers/storage/overlay/0a6251ac8edb01e4a1a495ea3104a4be3776a4bde7a9167c9216fe870b0b5a1d/diff /var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-87.ec2.internal_fbfcd267e5bdda32002f5b72a55d4954/haproxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229613 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323852    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.156441293s: [/var/lib/containers/storage/overlay/8e743a1c0c7cc9ef45cd9f86c51bdd235879c512aca4840e3a446be2f316bb6e/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr_f4ee5c50-5738-496a-b5ef-c86ead76e0f6/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229613 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323872    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.156458662s: [/var/lib/containers/storage/overlay/985ba12eb48566836f0fcad469e6eeb2711b77cf3eca5b054713e9dd9c612af3/diff /var/log/pods/openshift-ingress_kube-auth-proxy-78648788bf-npsk7_347c8613-0791-494e-83e3-faded5593d88/kube-auth-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229613 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323887    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058657162s: [/var/lib/containers/storage/overlay/38e2db64053a75d5e267a571b5c8029e880b65c662fc935cba8b4b78f3031bad/diff /var/log/pods/openshift-ingress_e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc_ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323900    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058645575s: [/var/lib/containers/storage/overlay/2850c96a313d433f759e56135d890d3c3b9d4452f3e507896aa256d912e71270/diff /var/log/pods/openshift-cluster-node-tuning-operator_tuned-n9fk9_f4b72e7f-cc4c-4662-a923-c184069aa118/tuned/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323918    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058659639s: [/var/lib/containers/storage/overlay/69e9e5b9009b44f81cb13f7e89161acc514055b4129ec8240b07d4f06c225a45/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-node-driver-registrar/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323931    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058672069s: [/var/lib/containers/storage/overlay/065839dc8c2c6a550d10b437944e397361cfc89bf9ccb63c18a6d705e7669644/diff /var/log/pods/openshift-multus_multus-7qzkz_f26fab1e-e6d4-4caa-bef6-81a09d78c0de/kube-multus/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323952    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058691709s: [/var/lib/containers/storage/overlay/f04694bfe7aed9c3a051f679dbed90340f776045c06d3bc39068d91ebc64b1d9/diff /var/log/pods/openshift-dns_dns-default-zmrdt_f59eba5d-a500-462e-9ecd-4ef9e553f873/dns/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323969    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058728969s: [/var/lib/containers/storage/overlay/6711a046696ca85a051e460f8eca0f62ee6736867b06bec3453afa502df3ff47/diff /var/log/pods/cert-manager_cert-manager-cainjector-5c57bbd4fc-9mjv9_9fe1660e-3733-4c7f-bc5c-3820d84aa143/cert-manager-cainjector/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.323986    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058724005s: [/var/lib/containers/storage/overlay/6da4544d5bf904910dc0072ef528e9149345d962762f71ecb9b0ce270fe91dc2/diff /var/log/pods/openshift-insights_insights-runtime-extractor-6bfcf_da9f1b66-0e1b-4c7e-a947-a897c00ccc19/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324000    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058737119s: [/var/lib/containers/storage/overlay/089a992d10746017d3c97ab20d1ddf9b955c8babb89ca5a1613bd5a1985f773b/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-m89vg_9257d59b-6aaf-423e-b7ed-2e64dbdfa535/openshift-state-metrics/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324022    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058748708s: [/var/lib/containers/storage/overlay/38cc698afa3d0a22ee2e64e3fa8b08e084747b51f63934648a8d8162d387223b/diff /var/log/pods/openshift-insights_insights-runtime-extractor-6bfcf_da9f1b66-0e1b-4c7e-a947-a897c00ccc19/exporter/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.229983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324036    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058771387s: [/var/lib/containers/storage/overlay/961cc1369cc5c2b2a8db1c37483931e3d284d8bf3daf0ea781ee526fa377a6a2/diff /var/log/pods/openshift-monitoring_metrics-server-7d987f75-ntp98_18d2ae0a-bf65-4aee-8730-093e00b54a24/metrics-server/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324049    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058782635s: [/var/lib/containers/storage/overlay/4c8d263b1f9b1e5f7f61488e19a80e3bdfd0a74fc133c4635b7c83f9383972d5/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm_dbd34da0-9b90-458d-9581-ed9caf3daa11/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324062    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.05883926s: [/var/lib/containers/storage/overlay/837e2e8c2d7c95b6015e40fc5ad8bcc348a2aaccbb5d63de3e44916d84e01aef/diff /var/log/pods/e2e-models-e2e-worker-w1-120394_distinct-2-w1-120394-kserve-766594cbd8-r7nr9_57a419b6-5333-4b2c-9f01-051dc82c8707/main/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324074    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058805768s: [/var/lib/containers/storage/overlay/6938a4b799445ead8836745535273347f3c908c027c27cae3317fe23a343f2cc/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/northd/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324088    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058826265s: [/var/lib/containers/storage/overlay/ffc640e61d9f2b8dc312d9f010b835c9e91e982e4f9cff0f120f887c46ef8f3a/diff /var/log/pods/openshift-ingress_e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb_551602c4-254b-4532-8961-a98bf0f81089/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230360 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324102    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058834563s: [/var/lib/containers/storage/overlay/fa8831aafda2d92f0ce3f59ac079b571f74f757924d7c3247d164db1d6711b3c/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j_e5ae7b61-cd5f-4a38-b038-28aaa6991eb7/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230360 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324118    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.685172521s: [/var/lib/containers/storage/overlay/735f0e10d9df130fc8a5a1c199498247b93c3829feaab94f9cf3d5136c7492bd/diff /var/log/pods/openshift-image-registry_image-registry-59767d8795-7vwkt_0db36d92-4712-4338-958a-ebabf2c137ba/registry/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230360 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324138    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058892908s: [/var/lib/containers/storage/overlay/ad9f3bb11ff9cab22055aad521309605cbe31ca829c1820e58022c92cded277f/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6_1c190514-ba33-4844-a9ec-e4bb47b5f706/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230360 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324155    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.691800539s: [/var/lib/containers/storage/overlay/e68ebf6860926a21a0cbfecd410ae12a4c05fb62dae7bcdbf219e8a050cbd79d/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8_e4c1132f-05b5-4664-84a2-ece57727b12a/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230516 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324169    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058879518s: [/var/lib/containers/storage/overlay/715bfa0d02500a7607ee5ba949c9655d2d098909dc187d3fa64c431aa173e55c/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb_cc682b81-5f9e-4cd1-98ae-d32e89878ea3/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230516 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324182    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058877501s: [/var/lib/containers/storage/overlay/4832bee5d30fb081704a3c96e43d22d7214d83e63b2bbc7c3a50de9c878d8550/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46_806cc8ce-f608-4975-94dd-5edade11fc8c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230516 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324196    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058888923s: [/var/lib/containers/storage/overlay/c032743b3f5164f05268a34d27f7a38a2df55f102c647c344a128377fc34aefd/diff /var/log/pods/openshift-ingress_e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx_81fd82a8-2bbf-43c8-962c-f8384287dc1f/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230516 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324212    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058959642s: [/var/lib/containers/storage/overlay/b569994ab3a7fd09901b9179b5a8ed81afe826237d0aeefc81429585cefd651a/diff /var/log/pods/e2e-models-e2e-worker-w1-120394_simulator-w1-120394-kserve-6979bf4f4f-scg5b_5734524d-67b2-4e24-a836-3ac078d5a271/main/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324235    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058944205s: [/var/lib/containers/storage/overlay/04144c7db0ef8494e8777526c23ed619a940be7d263e9bd174a3253037d6d77a/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d_8cba3353-fa51-41d3-acb4-a3cfbaae1daa/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324284    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058972507s: [/var/lib/containers/storage/overlay/25ec8cf788a39d4012f5db549bcc3d440d265141a6b16d25ff0974d692c4631b/diff /var/log/pods/openshift-ingress_e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8_6aa41ecd-4a93-4b17-8e84-3f3c018c4d21/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324314    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.058990389s: [/var/lib/containers/storage/overlay/861acfcf2321b99e26e85c3154bd9202cc22f4d667e5041f027bd7d24464ecb9/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-liveness-probe/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324330    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.059016346s: [/var/lib/containers/storage/overlay/48b947c743d65c0ae34868fbb47aeb890dde93159790d378edd81be829b75cfb/diff /var/log/pods/openshift-dns_node-resolver-hhsrb_ce8b7505-62a5-42f1-b432-d77200c21d2d/dns-node-resolver/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324343    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.059094936s: [/var/lib/containers/storage/overlay/d0ef0428e1dc75ad127901851f1b2fc725d21cd1b094959f49dcd7d557ab4f1a/diff /var/log/pods/openshift-ingress_e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths_5775b5a6-6407-46f4-84ef-96c708d7934b/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324356    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.059040945s: [/var/lib/containers/storage/overlay/1550c3e2e2a45dcfd84afdd33aa3690aa5dcb9b9475cb07cc27b0e0b34c41bb3/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/sbdb/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324369    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.059052639s: [/var/lib/containers/storage/overlay/e34d2a2f5077d6161aee9054ac97bc88dad0afe4dadc0770608b95d5aacf2f11/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/ovn-acl-logging/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.230847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324402    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.059082177s: [/var/lib/containers/storage/overlay/38f6c3a3bf1761682b345c03cb02c31f484b894296aec5d499ce5ae2e5601af5/diff /var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal_375359043409c490a8ed0d21b8afeaf5/kube-rbac-proxy-crio/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324423    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.059104476s: [/var/lib/containers/storage/overlay/1cfef9a8ec66cc7e72a2cc8f5db5f517a05984ada8feb9b6fcab0c85033fb6e5/diff /var/log/pods/openshift-multus_network-metrics-daemon-tk8ht_dd733778-7002-4a26-a4e8-4525d8f6bee9/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324439    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.059117647s: [/var/lib/containers/storage/overlay/4aaa9c0327050d64edf543a80049fd2fd46339540b39fe37be6c43a26d8f4b91/diff /var/log/pods/openshift-ingress_e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh_ef6443b3-b452-4868-9cde-f762cf522284/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324459    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.059137108s: [/var/lib/containers/storage/overlay/a143e700ec1dc2f1d0157b0b73e4385b467977554b146f0849b335957d53870f/diff /var/log/pods/openshift-ingress_e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7_73849b3d-673e-41a8-aff1-db9f0a9cce85/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324490    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.059182083s: [/var/lib/containers/storage/overlay/5c9ea955e524e35aff9b30ed78d97e370413ede7d4597334da2209e4500672c6/diff /var/log/pods/openshift-ingress_e2e-inf-a-872354-openshift-default-59df9df7-vnmfl_3c8d9150-66d1-4be8-8bca-3782fb0eac36/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231165 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324519    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.059203542s: [/var/lib/containers/storage/overlay/6890c8f8933a1e87b2636594da397d70dd586fa37081a11d1320e3554116bda7/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-vrvf2_07f874bb-122b-4794-ab59-736495e5cfdc/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231165 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324546    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.935888462s: [/var/lib/containers/storage/overlay/65d81a0f3b76a7c7a9475e4c40dceab09e6a9c9e98a2e2e3c4cc474c726a6216/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45_7e09749e-e648-40a3-a5f1-f0dc35fccf24/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231165 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324585    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.935921065s: [/var/lib/containers/storage/overlay/7b63acde62291120f32b79155fd4ac7703b60aa331f58870ab39e3003a8008d6/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb_cf85b137-2b1e-4e22-8f6e-744a02ea4e46/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231165 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324630    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.935968507s: [/var/lib/containers/storage/overlay/9012fabf6c9d37acb0fd084cebc4c8dd6d693a7cba8e42b02de88e893ac7cada/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324650    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.935987595s: [/var/lib/containers/storage/overlay/6df1c22d4bb429d567a6e21956e408dae481462cd854f2449182902dce8fee38/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg_96db7b02-b878-435a-869e-1d30e147c61d/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324667    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.936003222s: [/var/lib/containers/storage/overlay/7543a86d2f7dc71bebf4306a35f19682692a0213b5009a3a8a2a0250a1d74670/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/thanos-query/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324685    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.936018143s: [/var/lib/containers/storage/overlay/815d874b17f362f32c184d09c5fa4f25b24a51f559e41679c618ca076d1f092d/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy-metrics/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324710    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.93604105s: [/var/lib/containers/storage/overlay/d09ba0e972daf0879be4ba9795c180ae5ce98f9c793ff54605297f1c7dd5eb3d/diff /var/log/pods/openshift-ingress_e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88_cb09170f-1b12-4a26-8e5b-c5e2c2964a4d/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231468 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324727    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 21.925821018s: [/var/lib/containers/storage/overlay/1f335257c28f61349dc6c4495b958b214a44a29fd9b6b29ce082d223b3fa08f9/diff /var/log/pods/openshift-insights_insights-runtime-extractor-6bfcf_da9f1b66-0e1b-4c7e-a947-a897c00ccc19/extractor/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231468 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.324746    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.059491978s: [/var/lib/containers/storage/overlay/3b4eb99b514a85975e9e184eb65b4fc878deddd7205c795539ae51ac4de5caab/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj_bca2dabc-94e2-4875-ba3f-fa9332afe8dc/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231468 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.333748    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 49.693848315s: [/var/lib/containers/storage/overlay/b20116e8eeb7eef067efbcb5dd7bded5ed2f9d4621f36c8e5f282b97c52fd109/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v_6ebd3467-841a-4331-b149-8acf6e60c12c/maas-api/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:11:11.231468 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.369628    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m24.303106411s: [/var/lib/containers/storage/overlay/12431fb83ccd47188f1c3c13043aa6e2fae8d6f97a1eeb9704736aac6bb1d48b/diff /var/log/pods/opendatahub-operator-system_opendatahub-operator-controller-manager-76f76b79c-7n462_ec905c34-6dfb-483f-bb3d-7ba412262584/manager/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:11:11.231615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.399131    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.133869277s: [/var/lib/containers/storage/overlay/8e3943818fa8cc19ef6f82354867dfbe856e505e34cf4a7d05bc429843ee18e5/diff /var/log/pods/kuadrant-system_authorino-6f5b7cd5c7-brdvs_040ec9bf-319e-4bae-b233-4e4e7e74a379/authorino/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.231615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.400337    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"premium-w3-5eedab-kserve-self-signed-certs\""
Nov 02 18:11:11.231615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.411068    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-gclvz\""
Nov 02 18:11:11.231615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.415838    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.231615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.415920    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.231615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.416494    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 18:11:11.231615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.416801    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2c9b1d-dockercfg-tk2wk\""
Nov 02 18:11:11.231615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.416878    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-f0a3b8\""
Nov 02 18:11:11.231615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.419773    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.231963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.423845    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m2.372637211s: [/var/lib/containers/storage/overlay/42a7abc9d31e7e91afb1dc9f586b5fc298c1107b48f881228866c2735a90fee7/diff /var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-q2mbj_cae854c8-88f7-4201-b5d7-4e7c59fdf833/prometheus-operator-admission-webhook/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:11:11.231963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.425755    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.231963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.431928    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 02 18:11:11.231963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.432369    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5mdnhf4neu495\""
Nov 02 18:11:11.231963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.447916    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c4dbe3\""
Nov 02 18:11:11.231963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.453890    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m28.648934788s: [/var/lib/containers/storage/overlay/7957e0d146f8847395c58adcac4b12e2b0fbb5573d7580c6bc5eea896ab35073/diff /var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6jpq8_a1ea3dac-7073-4564-b247-89ea94a64342/manager/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:11:11.231963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.461497    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.196212098s: [/var/lib/containers/storage/overlay/1d679dc376e8b4b61ee5b35c347f887a08cc74c2c68502794ffcfd9744b157de/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq_5e54b0f0-5c3c-4441-a2a6-b452b9715e2e/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.232221 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.461740    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.196423661s: [/var/lib/containers/storage/overlay/83f8cdd611cedca899243d5a228c934d7754b14823c1219db9a0f8ced5da014e/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-5bqck_280a3e6c-c540-4c66-b963-c678ddca6e40/istio-proxy/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.232221 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.470159    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-2899ab\""
Nov 02 18:11:11.232221 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.496985    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.231708115s: [/var/lib/containers/storage/overlay/cb4bed28be09871a074b3642832880027d77099e69a0ae8241aeb94dab27fdaf/diff /var/log/pods/openshift-image-registry_node-ca-lhwgm_00b87b36-9ce2-480a-9924-22b33256fe1e/node-ca/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.232221 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.530165    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.232221 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.530165    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" podUID="e141d845-db9d-4f94-b05e-cfb7848e8fef" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.232410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.537994    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.232410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.538136    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.232410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.538321    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" podUID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.232410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.538671    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.232410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.539550    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j" podUID="e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.232599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.539732    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" podUID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.232599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.540115    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" podUID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.232599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.540446    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.232599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.540480    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.232599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.540509    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.232599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.540656    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.232921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.540685    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.232921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.541090    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.232921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.541127    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.232921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.545397    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f7th6\""
Nov 02 18:11:11.232921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.545765    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bb6d2e\""
Nov 02 18:11:11.232921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.546039    4456 reflector.go:446] "Caches populated" type="*v1.Service" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 18:11:11.232921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.546483    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.233186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.546663    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.281368666s: [/var/lib/containers/storage/overlay/c054c6b27f869c3385aeca4dc9bbc0f3c6a68aec5c08b24d9bfc20cfd651da17/diff /var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-xc9kg_3b0e614e-16b1-4e28-bea6-4b7b9a62b177/main/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.233186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.546676    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" podUID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.233186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.548064    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.233186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.548715    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.233186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.550634    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.233376 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.550669    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.233376 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.551024    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" podUID="d837dde2-3939-4400-8ea7-fee58c162983" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.233376 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.551919    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5bqck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.233376 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.551945    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.233376 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.554501    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5" podUID="02a2dbe6-baeb-40c4-be0f-835239be331f" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.233564 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.554665    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2" podUID="49238ac8-57fa-412a-88bd-681364a0b455" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.233564 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.556621    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" podUID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.233564 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.561439    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.233564 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.561481    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.233564 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.563277    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.233751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.565327    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.233751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.569455    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5knvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.233751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.569481    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx" podUID="8ef79a10-292b-42d9-aab0-53bee3e43ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.233751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.570515    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.233751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.570545    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.233957 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.587127    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.233957 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.588692    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.15:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.233957 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.588722    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/healthy\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.233957 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.591278    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.233957 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.592199    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.234146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.593634    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded"
Nov 02 18:11:11.234146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.593741    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vrvf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.234146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.593765    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.234146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.598273    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.234146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.598312    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.234340 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.598451    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.234340 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.598488    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.234340 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.599294    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.234340 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.599383    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.234340 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.599403    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.234529 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.599661    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.234529 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.599685    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.234529 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.612006    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m1.659773826s: [/var/lib/containers/storage/overlay/8d950223161f76ea6260ef66a4942c2c972122c5134abfc563f0de5c4bcace91/diff /var/log/pods/cert-manager_cert-manager-597b95cb84-hr8vz_e73be583-2a26-4379-8169-0a6b8784a11b/cert-manager-controller/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:11:11.234529 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.614342    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.34909247s: [/var/lib/containers/storage/overlay/8c0ca4c9e4e4e93687de6c7e703ba32222bc3330b9cf5dbd83a5cbdfbc1eb1a4/diff /var/log/pods/openshift-ingress_e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r_67b961eb-ab19-4d8d-bb2a-db7c94a93436/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.234684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.616081    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.350774368s: [/var/lib/containers/storage/overlay/61dba17c91911e3fe21e3760810cb0fdaa44600557f817d2a6d5f9e5af198421/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn_3d96e066-10c0-4fe2-9e24-ffd6195b22af/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.234684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.621242    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 02 18:11:11.234684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.621304    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-557feb\""
Nov 02 18:11:11.234684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.621327    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5oaelllrcvii0\""
Nov 02 18:11:11.234684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.628318    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-nv7lh\""
Nov 02 18:11:11.234684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.631247    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vqf9q\""
Nov 02 18:11:11.234684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.632794    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p2jm6\""
Nov 02 18:11:11.234684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.635040    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-b91583-dockercfg-vbwks\""
Nov 02 18:11:11.234684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.635088    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 18:11:11.234684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.635187    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-84fvd\""
Nov 02 18:11:11.235064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.635200    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 18:11:11.235064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.637595    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-f59572\""
Nov 02 18:11:11.235064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.641294    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 23.375981429s: [/var/lib/containers/storage/overlay/98ff010f77e7d96f5843be7a3a27066c1c197ffa6885b42fa9f473d20af37b2f/diff /var/log/pods/openshift-ingress_e2e-inf-a-872354-openshift-default-59df9df7-v5qhw_32802945-5e87-4339-a7eb-33212aab7ba5/istio-proxy/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.235064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.644495    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 18:11:11.235064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.644592    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"simulator-w5-863b87-kserve-self-signed-certs\""
Nov 02 18:11:11.235064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.646166    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 18:11:11.235064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.648850    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a29999-dockercfg-r7mt7\""
Nov 02 18:11:11.235064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.651187    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a5c1da-dockercfg-vlnwc\""
Nov 02 18:11:11.235064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.651462    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 02 18:11:11.235064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.655319    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-2899ab-dockercfg-lwvws\""
Nov 02 18:11:11.235420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.657910    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.235420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.659521    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 18:11:11.235420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.660751    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"default-dockercfg-h47km\""
Nov 02 18:11:11.235420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.663301    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fbbbf8-openshift-default-dockercfg-b9pj2\""
Nov 02 18:11:11.235420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.663555    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.235420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.669018    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.235420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.669043    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-863b87-openshift-default-dockercfg-5mwm4\""
Nov 02 18:11:11.235420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.669078    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 18:11:11.235420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.670942    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m1.221161497s: [/var/lib/containers/storage/overlay/2773d507773300fe9114d7cad1ed33c15472da65f5700864c7b4d36c59438123/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75_7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f/maas-api/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:11:11.235744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.672940    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 50.033039831s: [/var/lib/containers/storage/overlay/97e54822e3779b90c1439e95679938651a2f36f363a4df8db562bacd72f19b57/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2_97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922/payload-processing/0.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:11:11.235744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.673147    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.235744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.673165    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qbqb7\""
Nov 02 18:11:11.235744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.678062    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 59.510643352s: [/var/lib/containers/storage/overlay/1488f5a27e0831e0c877985c837f753313a612723997edaa573902b65e840bd5/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f_5ce3335c-36e6-4018-a264-5b79085f09f3/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.235744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.749051    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 18:11:11.235744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.753747    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"simulator-w3-5eedab-kserve-self-signed-certs\""
Nov 02 18:11:11.235744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.755691    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 18:11:11.236017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.781008    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m8.025140015s: [/var/lib/containers/storage/overlay/a2aa71b482739c7b0a0cc470b85d5d8947410f51100d4866f063fe123ede7d3e/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr_22b556c1-112b-49d4-932c-0143e912a54e/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.236017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.785530    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.787029    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.236017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.787069    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.790552    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236208 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.790550    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236208 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.798102    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236208 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.801044    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236208 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.802430    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236208 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.802719    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded"
Nov 02 18:11:11.236397 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.811348    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236397 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.813494    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.236397 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.813531    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236397 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.814618    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.236397 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.820893    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236582 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.822934    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.236582 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.822969    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236582 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.823935    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.236582 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.823963    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236582 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.827919    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.236776 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.827961    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236776 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.828903    4456 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.236776 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.828938    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.236776 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.828946    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236776 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.828972    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236989 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.830941    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.236989 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.830980    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236989 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.831178    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.236989 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.831233    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.236989 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.838922    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.237190 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.838927    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.134.0.28:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.237190 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.838956    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.237190 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.838955    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.237190 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.841904    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.237190 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.841932    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.237384 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.842949    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.237384 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.842979    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.237384 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.843312    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-069fa5-openshift-default-dockercfg-ckxtj\""
Nov 02 18:11:11.237384 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.848320    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.237384 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.849651    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.237384 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.849912    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.237615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.849946    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.237615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.854920    4456 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-pgmzg container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.237615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.854961    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.237615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.855944    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.237615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.858910    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.237910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.858954    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.237910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.859912    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.237910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.859947    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.237910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.860897    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.237910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.860907    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" podUID="44b66f41-99ae-446b-a50a-2ec0b71563a4" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.238111 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.860924    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.238111 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.860961    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" podUID="22b556c1-112b-49d4-932c-0143e912a54e" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.238111 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.862902    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.238111 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.862935    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.238111 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.862906    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.238312 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.862980    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.238312 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.865908    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.238312 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.865950    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.238312 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.870940    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.238312 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.870944    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.238506 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.870968    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.238506 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.874171    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.238506 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.875916    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.238506 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.880166    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.238506 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.887236    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.238699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.888910    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.238699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.892918    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.238699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.892950    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.238699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.895936    4456 patch_prober.go:29] interesting pod/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.238699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.895996    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.238937 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.896904    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.238937 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.896928    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.238937 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.901923    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.238937 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.903940    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.238937 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.903981    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.239135 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.905174    4456 trace.go:236] Trace[1328060412]: "Calculate volume metrics of istiod-ca-cert for pod openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" (02-Nov-2026 18:07:41.867) (total time: 1038ms):
Nov 02 18:11:11.239135 ip-10-0-141-87 kubenswrapper[4456]: Trace[1328060412]: [1.038069893s] [1.038069893s] END
Nov 02 18:11:11.239135 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.906914    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" podUID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.239135 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.909940    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.239135 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.909971    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.239135 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.911932    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.239362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.917908    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.239362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.917933    4456 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-pgmzg container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.239362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.917963    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.239362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.920916    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.239362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.920929    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" podUID="e4c1132f-05b5-4664-84a2-ece57727b12a" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.239362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.922924    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.239593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.924908    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.239593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:42.930321    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn2p2\""
Nov 02 18:11:11.239593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.079937    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 18:11:11.239593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.079979    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.239593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.079944    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 02 18:11:11.239593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.080171    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 18:11:11.239593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.080177    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 18:11:11.239593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.080205    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.239593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.080176    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 18:11:11.239593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.093439    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 18:11:11.239593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.093508    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.240049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.094137    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.240049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.094189    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 18:11:11.240049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.105719    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-96768b\""
Nov 02 18:11:11.240049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.107372    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a29999\""
Nov 02 18:11:11.240049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.107480    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.240049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.110868    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.240049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.118909    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-b91583\""
Nov 02 18:11:11.240049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.118923    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.240049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.121007    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 18:11:11.240049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.121040    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 18:11:11.240049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.121050    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-227939\""
Nov 02 18:11:11.240049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.121077    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.240761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.134673    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fbbbf8\""
Nov 02 18:11:11.240761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.134722    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 18:11:11.240761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.134921    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d5b905-openshift-default-dockercfg-vtgjv\""
Nov 02 18:11:11.240761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.134930    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rdt6t\""
Nov 02 18:11:11.240761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.134930    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qn5gh\""
Nov 02 18:11:11.240761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.135193    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"distinct-2-w1-120394-kserve-self-signed-certs\""
Nov 02 18:11:11.240761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.135301    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"distinct-w3-5eedab-kserve-self-signed-certs\""
Nov 02 18:11:11.240761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.144898    4456 trace.go:236] Trace[1320145552]: "Calculate volume metrics of tmp for pod openshift-console/downloads-6dd789c8b7-pgmzg" (02-Nov-2026 18:07:41.868) (total time: 1276ms):
Nov 02 18:11:11.240761 ip-10-0-141-87 kubenswrapper[4456]: Trace[1320145552]: [1.276538101s] [1.276538101s] END
Nov 02 18:11:11.240761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.148016    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2c9b1d-openshift-default-dockercfg-v7bnw\""
Nov 02 18:11:11.240761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.148082    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4679w\""
Nov 02 18:11:11.240761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.148345    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-96768b-dockercfg-dm54d\""
Nov 02 18:11:11.241234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.148473    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 18:11:11.241234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.148524    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mhkk4\""
Nov 02 18:11:11.241234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.148542    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9lngf\""
Nov 02 18:11:11.241234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.148587    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rj9zp\""
Nov 02 18:11:11.241234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.148593    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-j2fn5\""
Nov 02 18:11:11.241234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.148632    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fw7vt\""
Nov 02 18:11:11.241234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.148773    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-504de7-dockercfg-d4zk4\""
Nov 02 18:11:11.241234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.148891    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fbbbf8-dockercfg-29vsx\""
Nov 02 18:11:11.241234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.148958    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 18:11:11.241234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.161962    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2899ab-openshift-default-dockercfg-t6vss\""
Nov 02 18:11:11.241234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.162097    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mw4s9\""
Nov 02 18:11:11.241630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.162113    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-504de7-serving-cert\""
Nov 02 18:11:11.241630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.162200    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-96768b-openshift-default-dockercfg-v8qql\""
Nov 02 18:11:11.241630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.162225    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8a209f-openshift-default-dockercfg-whvdh\""
Nov 02 18:11:11.241630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.162239    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-872354-dockercfg-hw7xl\""
Nov 02 18:11:11.241630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.162319    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"distinct-2-w5-863b87-kserve-self-signed-certs\""
Nov 02 18:11:11.241630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.162336    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 18:11:11.241630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.162403    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ppwdb\""
Nov 02 18:11:11.241630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.162619    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 18:11:11.241630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.162623    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 18:11:11.241630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.244775    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.241630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.244783    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.241630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.244884    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.242075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.244964    4456 reflector.go:446] "Caches populated" type="*v1.CSIDriver" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 18:11:11.242075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.244983    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8a209f\""
Nov 02 18:11:11.242075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.255633    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m16.609699819s: [/var/lib/containers/storage/overlay/001563ee2fc791e222cb10a7b111cdb2621004a364564c35e4421a7c57d2db0b/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m_9051e10f-854f-4cc8-90a7-dda319c49c9a/maas-api/1.log]; will not log again for this container unless duration exceeds 2s
Nov 02 18:11:11.242075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.256908    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.242075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.258062    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.242075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.258095    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 02 18:11:11.242075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.271317    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 02 18:11:11.242075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.271340    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.242075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.271353    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.242075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.271751    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.242075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.285224    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-504de7\""
Nov 02 18:11:11.242473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.285231    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 18:11:11.242473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.285285    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.242473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.285474    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.242473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.285662    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 18:11:11.242473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.285660    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.242473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.285752    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.242473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.298675    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.242473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.298934    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 18:11:11.242473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.299428    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"default-dockercfg-8x5zb\""
Nov 02 18:11:11.242473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.299457    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-f0a3b8-serving-cert\""
Nov 02 18:11:11.242473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.299464    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bl7zd\""
Nov 02 18:11:11.242473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.299435    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-2672ed-serving-cert\""
Nov 02 18:11:11.242917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.299539    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"default-dockercfg-8dv9q\""
Nov 02 18:11:11.242917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.299637    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.242917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.299755    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"default-dockercfg-q6667\""
Nov 02 18:11:11.242917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.310462    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"premium-w1-120394-kserve-self-signed-certs\""
Nov 02 18:11:11.242917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.312074    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 18:11:11.242917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.312106    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a5c1da-openshift-default-dockercfg-gjtlc\""
Nov 02 18:11:11.242917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.313029    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"simulator-w1-120394-kserve-self-signed-certs\""
Nov 02 18:11:11.242917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.313089    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5fd3e0-openshift-default-dockercfg-kxlxh\""
Nov 02 18:11:11.242917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.313104    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qpmjh\""
Nov 02 18:11:11.242917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.313111    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4b9fl\""
Nov 02 18:11:11.242917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.313224    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 02 18:11:11.243304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.313250    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 18:11:11.243304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.313279    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 18:11:11.243304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.313279    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mnv5k\""
Nov 02 18:11:11.243304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.313308    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p5nzm\""
Nov 02 18:11:11.243304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.313283    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 02 18:11:11.243304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.313365    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 02 18:11:11.243304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.313380    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zt2z5\""
Nov 02 18:11:11.243304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.325541    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a29999-openshift-default-dockercfg-6b7bh\""
Nov 02 18:11:11.243304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.326877    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 18:11:11.243304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.326981    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 18:11:11.243304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.420342    4456 reflector.go:446] "Caches populated" type="*v1.Pod" reflector="pkg/kubelet/config/apiserver.go:66"
Nov 02 18:11:11.243304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.422114    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 18:11:11.243304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.422228    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.243759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.423104    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 18:11:11.243759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.423122    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 18:11:11.243759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.423223    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t2hrp\""
Nov 02 18:11:11.243759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.423246    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 18:11:11.243759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.423334    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 18:11:11.243759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.433599    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sn4p8\""
Nov 02 18:11:11.243759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.435681    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 18:11:11.243759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.436768    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gml5d\""
Nov 02 18:11:11.243759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.436785    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.243759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.436859    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-c810ca-openshift-default-dockercfg-s9n8v\""
Nov 02 18:11:11.243759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.437077    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 18:11:11.243759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.437080    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vjnnw\""
Nov 02 18:11:11.244270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.437204    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-872354\""
Nov 02 18:11:11.244270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.449322    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-557feb-openshift-default-dockercfg-lvqfg\""
Nov 02 18:11:11.244270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.450528    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.244270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.450552    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 02 18:11:11.244270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.450635    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 18:11:11.244270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.450685    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xdl26\""
Nov 02 18:11:11.244270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.450718    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.244270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.450732    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.244270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.450858    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 02 18:11:11.244270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.450864    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c4dbe3-dockercfg-qp9g8\""
Nov 02 18:11:11.244270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.450885    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2c9b1d\""
Nov 02 18:11:11.244270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.464234    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 18:11:11.244690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.464339    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.244690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.464340    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mwsx7\""
Nov 02 18:11:11.244690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.464461    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dklz6\""
Nov 02 18:11:11.244690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.464485    4456 reflector.go:446] "Caches populated" type="*v1.RuntimeClass" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 18:11:11.244690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.464533    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.244690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.464576    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 18:11:11.244690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.533661    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 18:11:11.244690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.534133    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 18:11:11.244690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.538397    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.244690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.545999    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 02 18:11:11.244690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.547078    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 02 18:11:11.245118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.547102    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a5c1da\""
Nov 02 18:11:11.245118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.547139    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 18:11:11.245118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.548315    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.245118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.558024    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8hdubeblopv5q\""
Nov 02 18:11:11.245118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.560482    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 02 18:11:11.245118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.560621    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.245118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.560640    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 18:11:11.245118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.563908    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.245118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.572914    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 18:11:11.245118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.572939    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 18:11:11.245496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.572919    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 18:11:11.245496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.574120    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 18:11:11.245496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.574147    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.245496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.574242    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.245496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.574318    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 18:11:11.245496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.574366    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 18:11:11.245496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.574391    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.245496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.574418    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.245496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.574670    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 18:11:11.245496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.587468    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.245496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.587707    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 18:11:11.245496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.587984    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 18:11:11.245950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.588097    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 18:11:11.245950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.588163    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 18:11:11.245950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.588284    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.245950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.588285    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.245950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.592482    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.245950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.593961    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded"
Nov 02 18:11:11.245950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.593970    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5bqck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.245950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.594044    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.246257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.599420    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.246257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.600455    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Liveness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.246257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.600486    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.246257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.601345    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.246257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.601513    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.246257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.601510    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jl8vw\""
Nov 02 18:11:11.246257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.601585    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 18:11:11.246257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.601596    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d5b905\""
Nov 02 18:11:11.246560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.602939    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.246560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.603002    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.246560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.610958    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5knvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.246560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.611013    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx" podUID="8ef79a10-292b-42d9-aab0-53bee3e43ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.246560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.611978    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.246750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.612053    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.246750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.614647    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8a209f-dockercfg-g8ztf\""
Nov 02 18:11:11.246750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.615680    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 18:11:11.246750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.616114    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-bfthl\""
Nov 02 18:11:11.246750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.616302    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 18:11:11.246750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.616306    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kqm5s\""
Nov 02 18:11:11.246750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.616468    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5nstj\""
Nov 02 18:11:11.246750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.616548    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-227939-dockercfg-4rtbn\""
Nov 02 18:11:11.246750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.616660    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-f0a3b8-dockercfg-h2cjr\""
Nov 02 18:11:11.246750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.616791    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-120394-openshift-default-dockercfg-qq496\""
Nov 02 18:11:11.247108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.616977    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-x9lxq\""
Nov 02 18:11:11.247108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.617052    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-557feb-dockercfg-nfpss\""
Nov 02 18:11:11.247108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.617120    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b2fc9f-openshift-default-dockercfg-tqz7r\""
Nov 02 18:11:11.247108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.617160    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"premium-w0-069fa5-kserve-self-signed-certs\""
Nov 02 18:11:11.247108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.617218    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5eedab-openshift-default-dockercfg-57c6m\""
Nov 02 18:11:11.247108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.617256    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.247108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.617337    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 18:11:11.247108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.629058    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-f59572-dockercfg-s5jw7\""
Nov 02 18:11:11.247108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.629261    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lkf8q\""
Nov 02 18:11:11.247108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.629451    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d5b905-dockercfg-sfklt\""
Nov 02 18:11:11.247108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.629910    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-872354-openshift-default-dockercfg-rdxd7\""
Nov 02 18:11:11.247359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.630535    4456 reflector.go:446] "Caches populated" type="*v1.Node" reflector="k8s.io/client-go/informers/factory.go:161"
Nov 02 18:11:11.247359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.630665    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 18:11:11.247359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.634950    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vrvf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.247359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.635028    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.247359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.639955    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.247359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:43.640018    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.247497 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.154287    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.247497 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.154388    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.247497 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.538862    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.247497 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.549183    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.247497 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.564678    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.247618 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.587668    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.247618 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.588805    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.247618 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.588908    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.247618 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.589178    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded"
Nov 02 18:11:11.247618 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.589216    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.247739 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.589438    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.247739 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.589641    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.247739 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.589854    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.247739 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.590090    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.247739 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.590113    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.247973 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.590137    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.247973 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.590364    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.247973 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.590418    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.247973 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.590465    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.247973 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.590767    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Liveness probe status=failure output="Get \"http://10.134.0.28:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.248167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.590788    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.248167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591043    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded"
Nov 02 18:11:11.248167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591077    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.248167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591105    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.248167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591122    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.248359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591160    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.248359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591182    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.248359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591207    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.248359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591227    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.248359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591280    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.248554 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591300    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.248554 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591595    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" podUID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.248554 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591777    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.248554 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591796    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.248554 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591920    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.248749 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591946    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.248749 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.591987    4456 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-pgmzg container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.248749 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.592006    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.248749 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.592115    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.248749 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.592638    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.248965 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.592739    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.248965 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.783248    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": context deadline exceeded"
Nov 02 18:11:11.248965 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.819068    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.248965 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.886033    4456 patch_prober.go:29] interesting pod/dns-default-zmrdt container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.248965 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.886099    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-zmrdt" podUID="f59eba5d-a500-462e-9ecd-4ef9e553f873" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.248965 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.908937    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.249207 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.909027    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.249207 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.922902    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.249207 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:44.922960    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.249207 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.475871    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 26.210538657s: [/var/lib/containers/storage/overlay/48c65928388ea29afd8ce97d9eec05ed39a6fd3da709a73d2174493d7947d9f4/diff /var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-c8vr7_08c9aa32-dadc-4112-b608-4cc64f56295c/kuadrant-console-plugin/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.249207 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.539551    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.249404 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.549648    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.249404 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.564838    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.249404 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.588245    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.249404 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.589622    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.249404 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.589657    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.249623 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.589635    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.249623 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.589784    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.249623 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.590013    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.249623 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.590252    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.249623 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.590558    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.249837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.592852    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.249837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.610662    4456 trace.go:236] Trace[1487526742]: "Calculate volume metrics of ca-trust-extracted for pod openshift-image-registry/image-registry-59767d8795-7vwkt" (02-Nov-2026 18:07:42.176) (total time: 3434ms):
Nov 02 18:11:11.249837 ip-10-0-141-87 kubenswrapper[4456]: Trace[1487526742]: [3.434105161s] [3.434105161s] END
Nov 02 18:11:11.249837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.746278    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.249837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.812970    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.249837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.848559    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.250079 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:45.848628    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.250079 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.539906    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.250079 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.565182    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.250079 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.588358    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.250079 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.590257    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.250079 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.590943    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.250338 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.611420    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.250338 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.630904    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.250338 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.630906    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.250338 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.632916    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.250338 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.634913    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.250574 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.652932    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.250574 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.654920    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.250574 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.788745    4456 patch_prober.go:29] interesting pod/image-registry-59767d8795-7vwkt container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.134.0.13:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.250574 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.788802    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" podUID="0db36d92-4712-4338-958a-ebabf2c137ba" containerName="registry" probeResult="failure" output="Get \"https://10.134.0.13:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.250574 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.789318    4456 patch_prober.go:29] interesting pod/image-registry-59767d8795-7vwkt container/registry namespace/openshift-image-registry: Liveness probe status=failure output="Get \"https://10.134.0.13:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.250830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.789357    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" podUID="0db36d92-4712-4338-958a-ebabf2c137ba" containerName="registry" probeResult="failure" output="Get \"https://10.134.0.13:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.250830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.801934    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.250830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.820284    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded" start-of-body=
Nov 02 18:11:11.250830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.820320    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded"
Nov 02 18:11:11.250830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.862953    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.251060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.887883    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.251060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.887900    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.251060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.916933    4456 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-tdx82 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.251060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:46.916977    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" podUID="0f2089f4-8ec7-45ce-a2df-0bf024b4158f" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.251060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:47.240960    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.251279 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:47.332932    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.251279 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:47.592775    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": context deadline exceeded"
Nov 02 18:11:11.251279 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:47.611900    4456 trace.go:236] Trace[67069988]: "iptables ChainExists" (02-Nov-2026 18:07:19.440) (total time: 28171ms):
Nov 02 18:11:11.251279 ip-10-0-141-87 kubenswrapper[4456]: Trace[67069988]: [28.17117014s] [28.17117014s] END
Nov 02 18:11:11.251279 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:47.611929    4456 trace.go:236] Trace[513385254]: "iptables ChainExists" (02-Nov-2026 18:06:11.410) (total time: 96201ms):
Nov 02 18:11:11.251279 ip-10-0-141-87 kubenswrapper[4456]: Trace[513385254]: [1m36.201119881s] [1m36.201119881s] END
Nov 02 18:11:11.251279 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:47.783549    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.251279 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:47.819718    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": context deadline exceeded"
Nov 02 18:11:11.251741 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:47.918973    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.251741 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:47.919024    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.251741 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:47.950947    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.251741 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:47.951001    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.251741 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:49.704952    4456 patch_prober.go:29] interesting pod/dns-default-zmrdt container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.251741 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:49.705009    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-zmrdt" podUID="f59eba5d-a500-462e-9ecd-4ef9e553f873" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.252693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:50.055930    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.252693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:50.848544    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": context deadline exceeded" start-of-body=
Nov 02 18:11:11.252693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:50.848619    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": context deadline exceeded"
Nov 02 18:11:11.252693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:51.346475    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.252693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:54.827930    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": context deadline exceeded"
Nov 02 18:11:11.252950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:07:57.211332    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.252950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:11.554888    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.252950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:12.525136    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" podUID="22b556c1-112b-49d4-932c-0143e912a54e" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.186:50052: i/o timeout"
Nov 02 18:11:11.252950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:12.962734    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.252950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:13.240211    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.134:50052: i/o timeout"
Nov 02 18:11:11.253164 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:15.265310    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded"
Nov 02 18:11:11.253164 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:15.697528    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5" podUID="02a2dbe6-baeb-40c4-be0f-835239be331f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.135:50052: i/o timeout"
Nov 02 18:11:11.253164 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:15.787454    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" podUID="e4c1132f-05b5-4664-84a2-ece57727b12a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.122:50052: i/o timeout"
Nov 02 18:11:11.253164 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:15.838192    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" podUID="d837dde2-3939-4400-8ea7-fee58c162983" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.116:50052: i/o timeout"
Nov 02 18:11:11.253164 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:15.862065    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.129:50052: i/o timeout"
Nov 02 18:11:11.253164 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:08.224934    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.253394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:10.449564    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.253394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:17.515235    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded"
Nov 02 18:11:11.253394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:10.457941    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.253394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:21.130046    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded"
Nov 02 18:11:11.253394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:23.089026    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.253603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:10.259692    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.253603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:25.474172    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.187:50052: i/o timeout"
Nov 02 18:11:11.253603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:27.881600    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.173:50052: i/o timeout"
Nov 02 18:11:11.253603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:09.294399    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.253603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:48.686282    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.253603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:48.854012    4456 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeNotReady"
Nov 02 18:11:11.255256 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:48.854054    4456 setters.go:546] "Node became not ready" node="ip-10-0-141-87.ec2.internal" condition={"type":"Ready","status":"False","lastHeartbeatTime":"2026-11-02T18:08:48Z","lastTransitionTime":"2026-11-02T18:08:48Z","reason":"KubeletNotReady","message":"PLEG is not healthy: pleg was last seen active 3m23.7722659s ago; threshold is 3m0s"}
Nov 02 18:11:11.255256 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:48.859927    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" podUID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.232:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.255256 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:48.874070    4456 ???:1] "http: TLS handshake error from 10.0.136.153:39912: EOF"
Nov 02 18:11:11.255256 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:48.876935    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.201:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.255256 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:48.957013    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.255256 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:48.957086    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.255487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:48.957739    4456 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-tdx82 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.255487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:48.957776    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" podUID="0f2089f4-8ec7-45ce-a2df-0bf024b4158f" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.255487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:48.963509    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" podUID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.190:50052: i/o timeout"
Nov 02 18:11:11.255487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:48.988657    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.201:50052: i/o timeout"
Nov 02 18:11:11.255487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.014952    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" podUID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.172:50052: i/o timeout"
Nov 02 18:11:11.255487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.016984    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.171:50052: i/o timeout"
Nov 02 18:11:11.255711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.038539    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.175:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.255711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.040483    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" podUID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.232:50052: i/o timeout"
Nov 02 18:11:11.255711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.045573    4456 trace.go:236] Trace[2026730208]: "Calculate volume metrics of istio-data for pod openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" (02-Nov-2026 18:08:47.303) (total time: 1742ms):
Nov 02 18:11:11.255711 ip-10-0-141-87 kubenswrapper[4456]: Trace[2026730208]: [1.742077284s] [1.742077284s] END
Nov 02 18:11:11.255711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.047946    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j" podUID="e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.44:50052: i/o timeout"
Nov 02 18:11:11.255711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.049066    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 02 18:11:11.255711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.049647    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.257377 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.058751    4456 ???:1] "http: TLS handshake error from 10.0.136.153:37332: EOF"
Nov 02 18:11:11.257377 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.093747    4456 logging.go:55] [core] [Channel #6429 SubChannel #6440]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.175:50052", ServerName: "10.134.0.175:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.175:50052: operation was canceled"
Nov 02 18:11:11.257377 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.096024    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" podUID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.45:50052: i/o timeout"
Nov 02 18:11:11.257377 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.098540    4456 logging.go:55] [core] [Channel #6435 SubChannel #6445]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.201:50052", ServerName: "10.134.0.201:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.201:50052: operation was canceled"
Nov 02 18:11:11.257377 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.098703    4456 logging.go:55] [core] [Channel #6430 SubChannel #6446]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.232:50052", ServerName: "10.134.0.232:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.232:50052: operation was canceled"
Nov 02 18:11:11.257377 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.104056    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:11.263324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:10.259650    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.263324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.113404    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.263324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.116888    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5" podUID="02a2dbe6-baeb-40c4-be0f-835239be331f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.135:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.263324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.163720    4456 patch_prober.go:29] interesting pod/dns-default-zmrdt container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.263324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.163776    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-zmrdt" podUID="f59eba5d-a500-462e-9ecd-4ef9e553f873" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.263545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:10.448512    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.263545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:10.449529    4456 patch_prober.go:29] interesting pod/metrics-server-7d987f75-ntp98 container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.16:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.263545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:11.274228    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.263545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:11.275375    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.263545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:11.554711    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.263754 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:11.885416    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.263754 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:12.525054    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.263754 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:12.684344    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.263754 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:12.684369    4456 patch_prober.go:29] interesting pod/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.263754 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:14.573591    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded" start-of-body=
Nov 02 18:11:11.263997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:14.689385    4456 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.263997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:14.689507    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.263997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:15.265241    4456 patch_prober.go:29] interesting pod/image-registry-59767d8795-7vwkt container/registry namespace/openshift-image-registry: Readiness probe status=failure output="Get \"https://10.134.0.13:5000/healthz\": context deadline exceeded" start-of-body=
Nov 02 18:11:11.263997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:15.696799    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.263997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:15.697240    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.263997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:15.697294    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.264241 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:15.755596    4456 patch_prober.go:29] interesting pod/image-registry-59767d8795-7vwkt container/registry namespace/openshift-image-registry: Liveness probe status=failure output="Get \"https://10.134.0.13:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.264241 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:17.515183    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.264241 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:17.515210    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.264241 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:21.802073    4456 patch_prober.go:29] interesting pod/network-check-target-nttxj container/network-check-target-container namespace/openshift-network-diagnostics: Readiness probe status=failure output="Get \"http://10.134.0.5:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.264241 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.175287    4456 logging.go:55] [core] [Channel #6438 SubChannel #6443]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.135:50052", ServerName: "10.134.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.135:50052: operation was canceled"
Nov 02 18:11:11.264442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.517444    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.264442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.517524    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.264442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.520261    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-7d987f75-ntp98" podUID="18d2ae0a-bf65-4aee-8730-093e00b54a24" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.16:10250/livez\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.264442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.520312    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.264442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.520243    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.264643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.520417    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.264643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.520475    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.264643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.520525    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.264643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.520577    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.264643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.520626    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.264875 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.520713    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.264875 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.520765    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded"
Nov 02 18:11:11.264875 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.520873    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.264875 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.520976    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.264875 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.521069    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" podUID="0db36d92-4712-4338-958a-ebabf2c137ba" containerName="registry" probeResult="failure" output="Get \"https://10.134.0.13:5000/healthz\": context deadline exceeded"
Nov 02 18:11:11.265077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.521164    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.265077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.521235    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.265077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.521289    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.265077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.521365    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" podUID="0db36d92-4712-4338-958a-ebabf2c137ba" containerName="registry" probeResult="failure" output="Get \"https://10.134.0.13:5000/healthz\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.265077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.521401    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.265281 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.521450    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.188:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.265281 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.521507    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d" containerName="network-check-target-container" probeResult="failure" output="Get \"http://10.134.0.5:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.265281 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.523101    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" podUID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.182:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.265281 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.524989    4456 logging.go:55] [core] [Channel #6428 SubChannel #6455]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.182:50052", ServerName: "10.134.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.182:50052: operation was canceled"
Nov 02 18:11:11.265281 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.535867    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.170:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.265482 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.537338    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.187:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.265482 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.537461    4456 logging.go:55] [core] [Channel #6439 SubChannel #6468]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.187:50052", ServerName: "10.134.0.187:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.187:50052: operation was canceled"
Nov 02 18:11:11.265482 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.537364    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" podUID="e141d845-db9d-4f94-b05e-cfb7848e8fef" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.180:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.265482 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.538068    4456 logging.go:55] [core] [Channel #6425 SubChannel #6463]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.180:50052", ServerName: "10.134.0.180:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.180:50052: operation was canceled"
Nov 02 18:11:11.265482 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.540050    4456 logging.go:55] [core] [Channel #6426 SubChannel #6464]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.170:50052", ServerName: "10.134.0.170:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.170:50052: operation was canceled"
Nov 02 18:11:11.265684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.540933    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" podUID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.183:50052: i/o timeout"
Nov 02 18:11:11.265684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.542030    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.170:50052: i/o timeout"
Nov 02 18:11:11.265684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.542390    4456 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-pgmzg container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.265684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.542421    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.265684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.545130    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.265684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.545179    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.266099 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.550426    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5bqck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.266099 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.550461    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.266099 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.551075    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.134:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.266099 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.551992    4456 logging.go:55] [core] [Channel #6481 SubChannel #6482]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.134:50052", ServerName: "10.134.0.134:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.134:50052: operation was canceled"
Nov 02 18:11:11.266099 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.555617    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" podUID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.266323 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.555933    4456 logging.go:55] [core] [Channel #6434 SubChannel #6486]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.45:50052", ServerName: "10.134.0.45:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.45:50052: operation was canceled"
Nov 02 18:11:11.266323 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.562417    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" podUID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.190:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.266323 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.562676    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.266323 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.562713    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.266323 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.562947    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.266526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.562997    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.266526 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.563076    4456 logging.go:55] [core] [Channel #6427 SubChannel #6490]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.190:50052", ServerName: "10.134.0.190:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.190:50052: operation was canceled"
Nov 02 18:11:11.266526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.563107    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.266526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.563891    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.266526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.564410    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.266732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.564446    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.266732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.568321    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.266732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.569920    4456 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-pgmzg container/download-server namespace/openshift-console: Liveness probe status=failure output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.266732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.569957    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.266732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.578441    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.266961 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.578471    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.266961 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.579305    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.129:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.266961 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.179542    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.266961 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.580213    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.266961 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.581035    4456 logging.go:55] [core] [Channel #6499 SubChannel #6500]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.129:50052", ServerName: "10.134.0.129:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.129:50052: operation was canceled"
Nov 02 18:11:11.267180 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.583219    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2" podUID="49238ac8-57fa-412a-88bd-681364a0b455" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.230:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.267180 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.583471    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" podUID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.183:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.267180 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.585031    4456 logging.go:55] [core] [Channel #6504 SubChannel #6505]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.230:50052", ServerName: "10.134.0.230:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.230:50052: operation was canceled"
Nov 02 18:11:11.267180 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.585069    4456 logging.go:55] [core] [Channel #6441 SubChannel #6506]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.183:50052", ServerName: "10.134.0.183:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.183:50052: operation was canceled"
Nov 02 18:11:11.267180 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.585146    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.173:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.267672 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.586849    4456 patch_prober.go:29] interesting pod/dns-default-zmrdt container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.134.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.267672 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.586882    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-zmrdt" podUID="f59eba5d-a500-462e-9ecd-4ef9e553f873" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.267672 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.587159    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kube-system/konnectivity-agent-mxsjq" podUID="367d8d85-a5e8-462d-b1d7-91a01e62b726" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.267672 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.588677    4456 logging.go:55] [core] [Channel #6431 SubChannel #6442]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.173:50052", ServerName: "10.134.0.173:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.173:50052: operation was canceled"
Nov 02 18:11:11.267672 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.589165    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.42:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.268116 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.592061    4456 logging.go:55] [core] [Channel #6516 SubChannel #6517]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.42:50052", ServerName: "10.134.0.42:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.42:50052: operation was canceled"
Nov 02 18:11:11.268116 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.605454    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.176:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.268116 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.606998    4456 logging.go:55] [core] [Channel #6521 SubChannel #6522]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.176:50052", ServerName: "10.134.0.176:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.176:50052: operation was canceled"
Nov 02 18:11:11.268116 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.607549    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" podUID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.172:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.268116 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.181999    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": context deadline exceeded"
Nov 02 18:11:11.268341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.608233    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.268341 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.609153    4456 logging.go:55] [core] [Channel #6424 SubChannel #6494]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.171:50052", ServerName: "10.134.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:56666->10.134.0.171:50052: use of closed network connection"
Nov 02 18:11:11.268341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.617605    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.200:50052: i/o timeout"
Nov 02 18:11:11.268341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.617772    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.200:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.268341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.617857    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" podUID="22b556c1-112b-49d4-932c-0143e912a54e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.186:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.268690 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.617977    4456 logging.go:55] [core] [Channel #6436 SubChannel #6530]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.200:50052", ServerName: "10.134.0.200:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.200:50052: operation was canceled"
Nov 02 18:11:11.268690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:15.697048    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.268690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.619491    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.268690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.619534    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.268690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.619477    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.43:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.269127 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:15.697165    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" podUID="e141d845-db9d-4f94-b05e-cfb7848e8fef" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.180:50052: i/o timeout"
Nov 02 18:11:11.269127 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.620709    4456 logging.go:55] [core] [Channel #6437 SubChannel #6531]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.186:50052", ServerName: "10.134.0.186:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.186:50052: operation was canceled"
Nov 02 18:11:11.269127 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.182311    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.42:50052: i/o timeout"
Nov 02 18:11:11.269127 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.621040    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.269127 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.621069    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.269766 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:15.787379    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" podUID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.182:50052: i/o timeout"
Nov 02 18:11:11.269766 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:15.979077    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.43:50052: i/o timeout"
Nov 02 18:11:11.269766 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.622121    4456 logging.go:55] [core] [Channel #6432 SubChannel #6536]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.43:50052", ServerName: "10.134.0.43:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.43:50052: operation was canceled"
Nov 02 18:11:11.269766 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.621801    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" podUID="44b66f41-99ae-446b-a50a-2ec0b71563a4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.189:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.269766 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.622208    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.270332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.622301    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.185:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.270332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:16.061454    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.270332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.622348    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.270332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.623188    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" podUID="e4c1132f-05b5-4664-84a2-ece57727b12a" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.270332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.625309    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-hr8vz" podUID="e73be583-2a26-4379-8169-0a6b8784a11b" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.134.0.26:9403/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.270654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:49.627327    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": context deadline exceeded"
Nov 02 18:11:11.270654 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:08:49.628043    4456 logging.go:55] [core] [Channel #6433 SubChannel #6542]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.189:50052", ServerName: "10.134.0.189:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.189:50052: operation was canceled"
Nov 02 18:11:11.270654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.924165    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.270654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.924319    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.270654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.924397    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": context deadline exceeded"
Nov 02 18:11:11.271106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.924540    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": context deadline exceeded"
Nov 02 18:11:11.271106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.924139    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded"
Nov 02 18:11:11.271106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.924773    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.271106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.924782    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" podUID="e141d845-db9d-4f94-b05e-cfb7848e8fef" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.180:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.271106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.924832    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.271462 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.924903    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2" podUID="49238ac8-57fa-412a-88bd-681364a0b455" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.230:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.271462 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.924911    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j" podUID="e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.44:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.271462 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.924982    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.271462 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.925038    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.271462 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.925322    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.134:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.272157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.925373    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.42:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.272157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.925432    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.272157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.925524    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.129:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.272157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.927528    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Nov 02 18:11:11.272157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.927541    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded"
Nov 02 18:11:11.272435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.927627    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.272435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.929210    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" podUID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.232:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.272435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.929312    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.201:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.272435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.930569    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.272435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.930632    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.272724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.931246    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.272724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.931302    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.272724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.931538    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.187:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.272724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.931635    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" podUID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.45:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.272724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.931707    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" podUID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.190:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.273142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.931801    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.173:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.273142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.932332    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5" podUID="02a2dbe6-baeb-40c4-be0f-835239be331f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.135:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.273142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.932470    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.273142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.932508    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.273142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.932743    4456 patch_prober.go:29] interesting pod/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.273445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.932732    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" podUID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.182:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.273445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.932774    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.216:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.273445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.932861    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.273445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.932883    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.273445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.932894    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" podUID="d837dde2-3939-4400-8ea7-fee58c162983" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.116:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.273745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.932939    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.273745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.932958    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.273745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.932997    4456 patch_prober.go:29] interesting pod/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.273745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.933023    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.57:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.273745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.933055    4456 patch_prober.go:29] interesting pod/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.274151 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.933078    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.176:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.274151 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.933081    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.274151 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.933329    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.274151 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.933390    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.274151 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.933556    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" podUID="22b556c1-112b-49d4-932c-0143e912a54e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.186:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.274409 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.933660    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" podUID="44b66f41-99ae-446b-a50a-2ec0b71563a4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.189:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.274409 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.934771    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.274409 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.934892    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.274409 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.935451    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.274409 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.935488    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.274880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.935588    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" podUID="c86c4f40-2978-40f0-bfc1-c3cfffca0d8c" containerName="ovnkube-controller" probeResult="failure" output="command timed out"
Nov 02 18:11:11.274880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.935599    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.274880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.935642    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.168:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.274880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.935714    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.274880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.935736    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" podUID="c86c4f40-2978-40f0-bfc1-c3cfffca0d8c" containerName="nbdb" probeResult="failure" output="command timed out"
Nov 02 18:11:11.274880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.935767    4456 patch_prober.go:29] interesting pod/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.275400 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.935789    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.73:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.275400 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.935736    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.199:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.275400 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.935854    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.275400 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.935883    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.275400 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.935895    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" podUID="c86c4f40-2978-40f0-bfc1-c3cfffca0d8c" containerName="sbdb" probeResult="failure" output="command timed out"
Nov 02 18:11:11.275400 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.935989    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="83deda5b-6ef6-4a66-a76c-888303e6e834" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 02 18:11:11.275896 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.936108    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.275896 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.936153    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-6bfcf" podUID="da9f1b66-0e1b-4c7e-a947-a897c00ccc19" containerName="extractor" probeResult="failure" output="command timed out"
Nov 02 18:11:11.275896 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.936153    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.275896 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.936119    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="83deda5b-6ef6-4a66-a76c-888303e6e834" containerName="prometheus" probeResult="failure" output="command timed out"
Nov 02 18:11:11.275896 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.939909    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.275896 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.939947    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.192:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.276380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.940135    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.276380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.940170    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.276380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.940708    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.276380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.940775    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.276380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.942492    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vrvf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.276722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.942542    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.276722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.943380    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded"
Nov 02 18:11:11.276722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.943832    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.276722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.944199    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 02 18:11:11.276722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.944377    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.277142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.944410    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.277142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.945210    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.200:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.277142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.945867    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.277142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.945907    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.277142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.947617    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" podUID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.277444 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.948054    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" podUID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.277444 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.949750    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.277444 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.950838    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.277444 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.968113    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.277444 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.969349    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.279619 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.969399    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.279619 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.970085    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.279619 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.971963    4456 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-pgmzg container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.279619 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.972013    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.279619 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.984982    4456 patch_prober.go:29] interesting pod/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.280285 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.985043    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.191:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.280285 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:50.988984    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerName="payload-pre-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.280285 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:51.051400    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 2.011771963s: [/var/lib/containers/storage/overlay/97e54822e3779b90c1439e95679938651a2f36f363a4df8db562bacd72f19b57/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2_97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922/payload-processing/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.280285 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:51.945105    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.280285 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:51.945167    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.280951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:51.958169    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.280951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:51.958228    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.280951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:51.968700    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.280951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:51.972431    4456 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-pgmzg container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.280951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:51.972482    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.280951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:52.673995    4456 scope.go:122] "RemoveContainer" containerID="6e1456e56a48150e86433f5c057ed17ac8273504ff8c6704b7f0bb813f92acd6"
Nov 02 18:11:11.281727 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:52.700034    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.281727 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:52.700112    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.281727 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:52.700926    4456 patch_prober.go:29] interesting pod/dns-default-zmrdt container/dns namespace/openshift-dns: Readiness probe status=failure output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.281727 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:52.701044    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-dns/dns-default-zmrdt" podUID="f59eba5d-a500-462e-9ecd-4ef9e553f873" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.7:8181/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.281727 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:53.101093    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 3.91685016s: [/var/lib/containers/storage/overlay/48c65928388ea29afd8ce97d9eec05ed39a6fd3da709a73d2174493d7947d9f4/diff /var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-c8vr7_08c9aa32-dadc-4112-b608-4cc64f56295c/kuadrant-console-plugin/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.281970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:53.876911    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-mxsjq" podUID="367d8d85-a5e8-462d-b1d7-91a01e62b726" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.281970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:53.998934    4456 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-tdx82 container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.281970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:53.998979    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" podUID="0f2089f4-8ec7-45ce-a2df-0bf024b4158f" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.111:8080/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.281970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:54.628922    4456 patch_prober.go:29] interesting pod/dns-default-zmrdt container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.134.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.281970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:54.628984    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-zmrdt" podUID="f59eba5d-a500-462e-9ecd-4ef9e553f873" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.281970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:54.999932    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-driver namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.282228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:54.999999    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-driver" probeResult="failure" output="Get \"http://127.0.0.1:10300/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.282228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:55.741970    4456 patch_prober.go:29] interesting pod/aws-ebs-csi-driver-node-6bb2n container/csi-node-driver-registrar namespace/openshift-cluster-csi-drivers: Liveness probe status=failure output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.282228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:55.742032    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-node-driver-registrar" probeResult="failure" output="Get \"http://127.0.0.1:10309/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.282228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:55.847526    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.282228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:08:55.847583    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.282445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:22.730583    4456 trace.go:236] Trace[659604256]: "Calculate volume metrics of crio-socket for pod openshift-insights/insights-runtime-extractor-6bfcf" (02-Nov-2026 18:08:56.056) (total time: 86673ms):
Nov 02 18:11:11.282445 ip-10-0-141-87 kubenswrapper[4456]: Trace[659604256]: [1m26.673663834s] [1m26.673663834s] END
Nov 02 18:11:11.282445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:23.695654    4456 trace.go:236] Trace[2122146250]: "Calculate volume metrics of host-var-lib-cni-networks-ovn-kubernetes for pod openshift-ovn-kubernetes/ovnkube-node-44l94" (02-Nov-2026 18:08:55.860) (total time: 87835ms):
Nov 02 18:11:11.282445 ip-10-0-141-87 kubenswrapper[4456]: Trace[2122146250]: [1m27.835133829s] [1m27.835133829s] END
Nov 02 18:11:11.282445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:23.695765    4456 trace.go:236] Trace[1237844311]: "Calculate volume metrics of hosts-file for pod openshift-dns/node-resolver-hhsrb" (02-Nov-2026 18:08:54.265) (total time: 89429ms):
Nov 02 18:11:11.282445 ip-10-0-141-87 kubenswrapper[4456]: Trace[1237844311]: [1m29.429773647s] [1m29.429773647s] END
Nov 02 18:11:11.282445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.315198    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.282445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.345810    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.282445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.539363    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.283007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.539437    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.283007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.540558    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.283007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.540619    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.283007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.540975    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.283007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.540999    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.283216 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.541285    4456 patch_prober.go:29] interesting pod/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.283216 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.541318    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.283216 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.541587    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.283216 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.541612    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.283216 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.541892    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.283415 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.541917    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.283415 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.711974    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.283415 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.712047    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.283415 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.743307    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.283415 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.743357    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.283702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.743614    4456 trace.go:236] Trace[1118005830]: "Calculate volume metrics of telemeter-trusted-ca-bundle for pod openshift-monitoring/telemeter-client-79c994cf5c-ckj5h" (02-Nov-2026 18:10:00.530) (total time: 24213ms):
Nov 02 18:11:11.283702 ip-10-0-141-87 kubenswrapper[4456]: Trace[1118005830]: [24.213254846s] [24.213254846s] END
Nov 02 18:11:11.283702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.743714    4456 trace.go:236] Trace[950339385]: "Calculate volume metrics of istio-envoy for pod openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" (02-Nov-2026 18:10:03.609) (total time: 21134ms):
Nov 02 18:11:11.283702 ip-10-0-141-87 kubenswrapper[4456]: Trace[950339385]: [21.134458776s] [21.134458776s] END
Nov 02 18:11:11.283702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.743904    4456 trace.go:236] Trace[1666446055]: "Calculate volume metrics of metrics-client-ca for pod openshift-monitoring/node-exporter-r45c5" (02-Nov-2026 18:10:02.800) (total time: 21942ms):
Nov 02 18:11:11.283702 ip-10-0-141-87 kubenswrapper[4456]: Trace[1666446055]: [21.942969985s] [21.942969985s] END
Nov 02 18:11:11.283702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.776141    4456 ???:1] "http: TLS handshake error from 10.0.136.153:41990: EOF"
Nov 02 18:11:11.283702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.776762    4456 ???:1] "http: TLS handshake error from 10.0.142.26:56714: EOF"
Nov 02 18:11:11.283702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.777410    4456 ???:1] "http: TLS handshake error from 10.0.142.26:39658: EOF"
Nov 02 18:11:11.283702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.779445    4456 trace.go:236] Trace[329421399]: "Calculate volume metrics of istio-podinfo for pod openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" (02-Nov-2026 18:10:07.122) (total time: 17657ms):
Nov 02 18:11:11.283702 ip-10-0-141-87 kubenswrapper[4456]: Trace[329421399]: [17.657117909s] [17.657117909s] END
Nov 02 18:11:11.283702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.780127    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded"
Nov 02 18:11:11.284223 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.800275    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-mxsjq" podUID="367d8d85-a5e8-462d-b1d7-91a01e62b726" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.284223 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.860604    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" podUID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.190:50052: i/o timeout"
Nov 02 18:11:11.284223 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.864863    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.171:50052: i/o timeout"
Nov 02 18:11:11.284223 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.865205    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" podUID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.232:50052: i/o timeout"
Nov 02 18:11:11.284223 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.865889    4456 trace.go:236] Trace[2114351833]: "Calculate volume metrics of kube-api-access-k85vn for pod openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" (02-Nov-2026 18:09:34.213) (total time: 50652ms):
Nov 02 18:11:11.284223 ip-10-0-141-87 kubenswrapper[4456]: Trace[2114351833]: [50.652513725s] [50.652513725s] END
Nov 02 18:11:11.284223 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.204560    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5" podUID="02a2dbe6-baeb-40c4-be0f-835239be331f" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.135:50052: i/o timeout"
Nov 02 18:11:11.284510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.206923    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.187:50052: i/o timeout"
Nov 02 18:11:11.284510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.207137    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.173:50052: i/o timeout"
Nov 02 18:11:11.284510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.207275    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.129:50052: i/o timeout"
Nov 02 18:11:11.284510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.223009    4456 trace.go:236] Trace[838450740]: "Calculate volume metrics of kube-api-access-kjmhh for pod openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2" (02-Nov-2026 18:10:22.677) (total time: 36545ms):
Nov 02 18:11:11.284510 ip-10-0-141-87 kubenswrapper[4456]: Trace[838450740]: [36.545162524s] [36.545162524s] END
Nov 02 18:11:11.284510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.223115    4456 trace.go:236] Trace[2116591221]: "Calculate volume metrics of data-volume for pod openshift-insights/insights-runtime-extractor-6bfcf" (02-Nov-2026 18:10:22.730) (total time: 36492ms):
Nov 02 18:11:11.284510 ip-10-0-141-87 kubenswrapper[4456]: Trace[2116591221]: [36.492486215s] [36.492486215s] END
Nov 02 18:11:11.284510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.223148    4456 trace.go:236] Trace[135062551]: "Calculate volume metrics of istio-podinfo for pod openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" (02-Nov-2026 18:10:22.929) (total time: 36293ms):
Nov 02 18:11:11.284510 ip-10-0-141-87 kubenswrapper[4456]: Trace[135062551]: [36.293863845s] [36.293863845s] END
Nov 02 18:11:11.284510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.223401    4456 trace.go:236] Trace[946286985]: "Calculate volume metrics of workload-socket for pod openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" (02-Nov-2026 18:10:22.935) (total time: 36287ms):
Nov 02 18:11:11.284936 ip-10-0-141-87 kubenswrapper[4456]: Trace[946286985]: [36.287628138s] [36.287628138s] END
Nov 02 18:11:11.284936 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.223641    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" podUID="e4c1132f-05b5-4664-84a2-ece57727b12a" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.122:50052: i/o timeout"
Nov 02 18:11:11.284936 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.223849    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" podUID="e141d845-db9d-4f94-b05e-cfb7848e8fef" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.180:50052: i/o timeout"
Nov 02 18:11:11.284936 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.224018    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.43:50052: i/o timeout"
Nov 02 18:11:11.284936 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.224269    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.42:50052: i/o timeout"
Nov 02 18:11:11.284936 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.224459    4456 trace.go:236] Trace[32406477]: "Calculate volume metrics of local-certs for pod openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" (02-Nov-2026 18:10:23.711) (total time: 35513ms):
Nov 02 18:11:11.284936 ip-10-0-141-87 kubenswrapper[4456]: Trace[32406477]: [35.5133225s] [35.5133225s] END
Nov 02 18:11:11.284936 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.224508    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.285276 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.224536    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.285276 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.224587    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.285276 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.224612    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.285276 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.224711    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" podUID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.45:50052: i/o timeout"
Nov 02 18:11:11.285276 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.224871    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.200:50052: i/o timeout"
Nov 02 18:11:11.285276 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.225998    4456 trace.go:236] Trace[371182773]: "Calculate volume metrics of bound-sa-token for pod cert-manager/cert-manager-597b95cb84-hr8vz" (02-Nov-2026 18:10:22.378) (total time: 36847ms):
Nov 02 18:11:11.285276 ip-10-0-141-87 kubenswrapper[4456]: Trace[371182773]: [36.847470569s] [36.847470569s] END
Nov 02 18:11:11.285605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.270935    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded" start-of-body=
Nov 02 18:11:11.285605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.226078    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded"
Nov 02 18:11:11.285605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.295887    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.285605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.226189    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.285605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.227157    4456 trace.go:236] Trace[1100068801]: "Calculate volume metrics of model-cache for pod e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" (02-Nov-2026 18:10:23.714) (total time: 35512ms):
Nov 02 18:11:11.285605 ip-10-0-141-87 kubenswrapper[4456]: Trace[1100068801]: [35.512950441s] [35.512950441s] END
Nov 02 18:11:11.285605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.227338    4456 trace.go:236] Trace[2058437877]: "Calculate volume metrics of tmp-dir for pod openshift-dns/node-resolver-hhsrb" (02-Nov-2026 18:10:23.695) (total time: 35531ms):
Nov 02 18:11:11.285605 ip-10-0-141-87 kubenswrapper[4456]: Trace[2058437877]: [35.531544948s] [35.531544948s] END
Nov 02 18:11:11.290979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.227370    4456 trace.go:236] Trace[1018517338]: "Calculate volume metrics of istio-data for pod openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" (02-Nov-2026 18:10:23.722) (total time: 35504ms):
Nov 02 18:11:11.290979 ip-10-0-141-87 kubenswrapper[4456]: Trace[1018517338]: [35.504769552s] [35.504769552s] END
Nov 02 18:11:11.290979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.227580    4456 trace.go:236] Trace[940501015]: "Calculate volume metrics of istio-data for pod openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" (02-Nov-2026 18:10:23.772) (total time: 35455ms):
Nov 02 18:11:11.290979 ip-10-0-141-87 kubenswrapper[4456]: Trace[940501015]: [35.455036246s] [35.455036246s] END
Nov 02 18:11:11.290979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.472696    4456 trace.go:236] Trace[1098542804]: "iptables ChainExists" (02-Nov-2026 18:08:48.973) (total time: 130498ms):
Nov 02 18:11:11.290979 ip-10-0-141-87 kubenswrapper[4456]: Trace[1098542804]: [2m10.498987309s] [2m10.498987309s] END
Nov 02 18:11:11.290979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.472902    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.134:50052: i/o timeout"
Nov 02 18:11:11.290979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.473168    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.290979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.473161    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/kube-apiserver-proxy-ip-10-0-141-87.ec2.internal" podUID="fbfcd267e5bdda32002f5b72a55d4954" containerName="haproxy" probeResult="failure" output="Get \"https://172.20.0.1:6443/version\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.291371 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.473249    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.291371 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.473447    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.291371 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.473542    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.291371 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.473776    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.291371 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.473980    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.291577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.474124    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": context deadline exceeded"
Nov 02 18:11:11.291577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.474185    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.291577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.474327    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded"
Nov 02 18:11:11.291577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.474540    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" podUID="d837dde2-3939-4400-8ea7-fee58c162983" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.116:50052: i/o timeout"
Nov 02 18:11:11.291577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.474551    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-mxsjq" podUID="367d8d85-a5e8-462d-b1d7-91a01e62b726" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.291577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.474789    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.291852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.515275    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.291852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.515962    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.291852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.626887    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.201:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.291852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.474137    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.291852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.627467    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.292069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.627486    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.292069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.627537    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.193:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.292069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.628271    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" podUID="d837dde2-3939-4400-8ea7-fee58c162983" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.116:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.292069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.629321    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j" podUID="e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.44:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.292069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.629554    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" podUID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.182:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.292275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.474157    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5bqck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.292275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.629604    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.292275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.474531    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.292275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.629680    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.292275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.491319    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.292480 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.629757    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.292480 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.491341    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.292480 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.629845    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.292480 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.491360    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.292480 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.629892    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.292687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.491373    4456 patch_prober.go:29] interesting pod/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.292687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.629952    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.292687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.491392    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.292687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.629999    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.292687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.491452    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.292908 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:10:59.630113    4456 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" containerID="d8f0d9bcc31bfb03128cf8096922bcabc511ffd73d885355f5b2ae417aa2bb93" cmd=["test","-f","/etc/cni/net.d/10-ovn-kubernetes.conf"]
Nov 02 18:11:11.292908 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.630154    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" podUID="c86c4f40-2978-40f0-bfc1-c3cfffca0d8c" containerName="ovnkube-controller" probeResult="failure" output="command timed out: \"test -f /etc/cni/net.d/10-ovn-kubernetes.conf\" timed out after 1s"
Nov 02 18:11:11.292908 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:10:59.630239    4456 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" containerID="bd6105fca3203892e8cc26e61c1ae82d5d6b228d6358c7367e450783ccbc3ce9" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi"]
Nov 02 18:11:11.292908 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.630265    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-k8s-0" podUID="83deda5b-6ef6-4a66-a76c-888303e6e834" containerName="prometheus" probeResult="failure" output="command timed out: \"sh -c if [ -x \\\"$(command -v curl)\\\" ]; then exec curl --fail http://localhost:9090/-/ready; elif [ -x \\\"$(command -v wget)\\\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/ready; else exit 1; fi\" timed out after 3s"
Nov 02 18:11:11.292908 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:10:59.630406    4456 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" containerID="bd6105fca3203892e8cc26e61c1ae82d5d6b228d6358c7367e450783ccbc3ce9" cmd=["sh","-c","if [ -x \"$(command -v curl)\" ]; then exec curl --fail http://localhost:9090/-/healthy; elif [ -x \"$(command -v wget)\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/healthy; else exit 1; fi"]
Nov 02 18:11:11.293130 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.630445    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-monitoring/prometheus-k8s-0" podUID="83deda5b-6ef6-4a66-a76c-888303e6e834" containerName="prometheus" probeResult="failure" output="command timed out: \"sh -c if [ -x \\\"$(command -v curl)\\\" ]; then exec curl --fail http://localhost:9090/-/healthy; elif [ -x \\\"$(command -v wget)\\\" ]; then exec wget -q -O /dev/null http://localhost:9090/-/healthy; else exit 1; fi\" timed out after 3s"
Nov 02 18:11:11.293130 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:10:59.631963    4456 logging.go:55] [core] [Channel #6612 SubChannel #6627]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.116:50052", ServerName: "10.134.0.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.116:50052: operation was canceled"
Nov 02 18:11:11.293130 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.632223    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.293130 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.632244    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.200:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.293130 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.633511    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.293349 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.640691    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" podUID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.232:50052: i/o timeout"
Nov 02 18:11:11.293349 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.640967    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.171:50052: i/o timeout"
Nov 02 18:11:11.293349 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.644238    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" podUID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.190:50052: i/o timeout"
Nov 02 18:11:11.293349 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.644452    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2" podUID="49238ac8-57fa-412a-88bd-681364a0b455" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.230:50052: i/o timeout"
Nov 02 18:11:11.293349 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.645487    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.293349 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.646215    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.293594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.646375    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j" podUID="e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.44:50052: i/o timeout"
Nov 02 18:11:11.293594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.646755    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.201:50052: i/o timeout"
Nov 02 18:11:11.293594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.647323    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.293594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.647363    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.293594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.647420    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" podUID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.182:50052: i/o timeout"
Nov 02 18:11:11.293594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:24.878688    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.293869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.650380    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.293869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.651626    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" podUID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.183:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.293869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.651942    4456 patch_prober.go:29] interesting pod/network-check-target-nttxj container/network-check-target-container namespace/openshift-network-diagnostics: Readiness probe status=failure output="Get \"http://10.134.0.5:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.293869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.651989    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-network-diagnostics/network-check-target-nttxj" podUID="a974c7e3-bf97-4575-bec1-c09fcd018f0d" containerName="network-check-target-container" probeResult="failure" output="Get \"http://10.134.0.5:8080/\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.293869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.651988    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.176:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.294090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.653186    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-insights/insights-runtime-extractor-6bfcf" podUID="da9f1b66-0e1b-4c7e-a947-a897c00ccc19" containerName="extractor" probeResult="failure" output="command timed out"
Nov 02 18:11:11.294090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.657478    4456 trace.go:236] Trace[1528763978]: "iptables ChainExists" (02-Nov-2026 18:08:48.967) (total time: 130690ms):
Nov 02 18:11:11.294090 ip-10-0-141-87 kubenswrapper[4456]: Trace[1528763978]: [2m10.690271494s] [2m10.690271494s] END
Nov 02 18:11:11.294090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.658876    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2" podUID="49238ac8-57fa-412a-88bd-681364a0b455" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.230:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.294090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.491411    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.294090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.659170    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" podUID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.190:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.294090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.659212    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.294358 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.659932    4456 patch_prober.go:29] interesting pod/dns-default-zmrdt container/dns namespace/openshift-dns: Liveness probe status=failure output="Get \"http://10.134.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.294358 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.659958    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-dns/dns-default-zmrdt" podUID="f59eba5d-a500-462e-9ecd-4ef9e553f873" containerName="dns" probeResult="failure" output="Get \"http://10.134.0.7:8080/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.294358 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.701189    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" podUID="e141d845-db9d-4f94-b05e-cfb7848e8fef" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.180:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.294358 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.708596    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.134:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.294358 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.740955    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.175:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.294567 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:10:59.763876    4456 logging.go:55] [core] [Channel #6623 SubChannel #6632]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.182:50052", ServerName: "10.134.0.182:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.182:50052: operation was canceled"
Nov 02 18:11:11.294567 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:10:59.765064    4456 logging.go:55] [core] [Channel #6616 SubChannel #6617]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.183:50052", ServerName: "10.134.0.183:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.183:50052: operation was canceled"
Nov 02 18:11:11.294567 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:10:59.766993    4456 logging.go:55] [core] [Channel #6625 SubChannel #6633]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.200:50052", ServerName: "10.134.0.200:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.200:50052: operation was canceled"
Nov 02 18:11:11.294567 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:10:59.767949    4456 logging.go:55] [core] [Channel #6622 SubChannel #6631]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.44:50052", ServerName: "10.134.0.44:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.44:50052: operation was canceled"
Nov 02 18:11:11.294567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.768264    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" podUID="c86c4f40-2978-40f0-bfc1-c3cfffca0d8c" containerName="nbdb" probeResult="failure" output="command timed out"
Nov 02 18:11:11.294787 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.789941    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.170:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.294787 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.790974    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5" podUID="02a2dbe6-baeb-40c4-be0f-835239be331f" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.135:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.294787 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:10:59.842969    4456 logging.go:55] [core] [Channel #6608 SubChannel #6645]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.190:50052", ServerName: "10.134.0.190:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.190:50052: operation was canceled"
Nov 02 18:11:11.294787 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:10:59.849909    4456 logging.go:55] [core] [Channel #6607 SubChannel #6644]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.230:50052", ServerName: "10.134.0.230:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.230:50052: operation was canceled"
Nov 02 18:11:11.294787 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:10:59.858729    4456 logging.go:55] [core] [Channel #6606 SubChannel #6648]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.134:50052", ServerName: "10.134.0.134:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.134:50052: operation was canceled"
Nov 02 18:11:11.295002 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:10:59.865735    4456 logging.go:55] [core] [Channel #6611 SubChannel #6626]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.135:50052", ServerName: "10.134.0.135:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.135:50052: operation was canceled"
Nov 02 18:11:11.295002 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:10:59.868615    4456 logging.go:55] [core] [Channel #6624 SubChannel #6636]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.180:50052", ServerName: "10.134.0.180:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.180:50052: operation was canceled"
Nov 02 18:11:11.295002 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.878017    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" podUID="44b66f41-99ae-446b-a50a-2ec0b71563a4" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.189:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.295002 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:10:59.880474    4456 logging.go:55] [core] [Channel #6640 SubChannel #6643]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.176:50052", ServerName: "10.134.0.176:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.176:50052: operation was canceled"
Nov 02 18:11:11.295002 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:10:59.881003    4456 logging.go:55] [core] [Channel #6609 SubChannel #6653]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.175:50052", ServerName: "10.134.0.175:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.175:50052: operation was canceled"
Nov 02 18:11:11.295222 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:10:59.887449    4456 logging.go:55] [core] [Channel #6613 SubChannel #6650]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.189:50052", ServerName: "10.134.0.189:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.189:50052: operation was canceled"
Nov 02 18:11:11.295222 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:10:59.892612    4456 log.go:32] "ExecSync cmd from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" containerID="c85e2174c667b93c00a4c61bce54d357ace38a16f087c31c6415b9ccbc761f92" cmd=["/bin/bash","-c","set -xeo pipefail\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\novndb-readiness-probe \"sb\"\n"]
Nov 02 18:11:11.295222 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.892653    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ovn-kubernetes/ovnkube-node-44l94" podUID="c86c4f40-2978-40f0-bfc1-c3cfffca0d8c" containerName="sbdb" probeResult="failure" output="command timed out: \"/bin/bash -c set -xeo pipefail\\n. /ovnkube-lib/ovnkube-lib.sh || exit 1\\novndb-readiness-probe \\\"sb\\\"\\n\" timed out after 5s"
Nov 02 18:11:11.295222 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:10:59.892567    4456 log.go:32] "ListContainers with filter from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded" filter=""
Nov 02 18:11:11.295222 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:10:59.893143    4456 kuberuntime_container.go:554] "ListContainers failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
Nov 02 18:11:11.295222 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:10:59.893166    4456 generic.go:254] "GenericPLEG: Unable to retrieve pods" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
Nov 02 18:11:11.295450 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:10:59.893272    4456 logging.go:55] [core] [Channel #6610 SubChannel #6649]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.170:50052", ServerName: "10.134.0.170:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.170:50052: operation was canceled"
Nov 02 18:11:11.295450 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.897497    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vrvf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.295450 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.897545    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.295450 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.898990    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.295450 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.499777    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": context deadline exceeded" start-of-body=
Nov 02 18:11:11.295655 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.899101    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": context deadline exceeded"
Nov 02 18:11:11.295655 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.509938    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.295655 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.899148    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.295655 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.626748    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.295655 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:10:59.899257    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.295655 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:00.087261    4456 log.go:32] "Failed when writing line to log file" err="client disconnected" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" line={}
Nov 02 18:11:11.295921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.088837    4456 ???:1] "http: TLS handshake error from 10.0.141.87:45524: EOF"
Nov 02 18:11:11.295921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.091672    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" podUID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.232:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.295921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.242735    4456 patch_prober.go:29] interesting pod/prometheus-operator-admission-webhook-6489d8bf89-q2mbj container/prometheus-operator-admission-webhook namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.295921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.242795    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" probeResult="failure" output="Get \"https://10.134.0.11:8443/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.295921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.242861    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.925552692s: [/var/lib/containers/storage/overlay/b01ca7c42e592661866b33c6e00f21a35920d215fcb234779a7e876cf886a10f/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.295921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.242944    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.925658809s: [/var/lib/containers/storage/overlay/a5abae015c3465904cc5c7174a5b19880c528412ca15a33a710b819431e1c2f7/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.296179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.266035    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.296179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.266097    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.179:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.296179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.270952    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.296179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.271011    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.227:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.296179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.273111    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.296380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.273186    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.225:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.296380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.355933    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.967907576s: [/var/lib/containers/storage/overlay/491769e17d91cf9015f229933a02ba515455f7a916be79b9866f14aaeb3a54d7/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.296380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.355781    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.967748326s: [/var/lib/containers/storage/overlay/2848def6302080170b0488e9cb4ed92ffb69bd4f38ed3e4eba1e06c02f7c02fa/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.296380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.355990    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.967964995s: [/var/lib/containers/storage/overlay/02d4c961dc5c71ea22ce464a653f70d8ea7afa2333251e6e0f32faecb297ca22/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.296380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.356040    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.968013688s: [/var/lib/containers/storage/overlay/5eda4bcefdda7ae7749489bd6e8252ea64d62d114e01e22912d64207986209b0/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.296380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.476115    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" probeResult="failure" output="Get \"https://10.134.0.76:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.296634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.476252    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.296634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.476383    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.296634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.476493    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" probeResult="failure" output="Get \"https://10.134.0.48:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.296634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.476701    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k" podUID="14d43510-cfbb-4ddc-88cb-ccd4cb3a543d" containerName="main" probeResult="failure" output="Get \"https://10.134.0.31:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.296634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.476725    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" probeResult="failure" output="Get \"https://10.134.0.49:8000/health\": context deadline exceeded"
Nov 02 18:11:11.296857 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.476783    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/health\": context deadline exceeded"
Nov 02 18:11:11.296857 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.476791    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": context deadline exceeded"
Nov 02 18:11:11.296857 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.476851    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded"
Nov 02 18:11:11.296857 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.525089    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.296857 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.528976    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" probeResult="failure" output="Get \"https://10.134.0.77:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.297071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.529053    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" probeResult="failure" output="Get \"https://10.134.0.51:8000/health\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.297071 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:00.573884    4456 log.go:32] "Status from runtime service failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
Nov 02 18:11:11.297071 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:00.573934    4456 kubelet.go:3135] "Container runtime sanity check failed" err="rpc error: code = DeadlineExceeded desc = context deadline exceeded"
Nov 02 18:11:11.297071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.573115    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.185078064s: [/var/lib/containers/storage/overlay/6db68c524c0be1eea698df87a8b9a3de739d2a7cafcb2c4289aec09a3b1cfe4f/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600327    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212291737s: [/var/lib/containers/storage/overlay/48b0c974b21e6e5c0615a44fe7b2a01a2a0049cec1284df8211410724ae86bc8/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600360    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212330416s: [/var/lib/containers/storage/overlay/76420521e8040790a74114ea6b6329940e29b9c3789f480a932d32ff1a407957/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600387    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212355705s: [/var/lib/containers/storage/overlay/7f6de02a430f8b7e33a65f8a46c453dc84b8b909c0cab1337c53ba4b1fe6ebc3/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600413    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212379477s: [/var/lib/containers/storage/overlay/470e14ce534cc9ccf01ebec836415ae3fdf34ea27e170f57ace24876fb78fb58/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.297356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600442    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212406253s: [/var/lib/containers/storage/overlay/571669d5d66ff820dd233ab868967067d48bbd1c815de34fbc4a6936fddd205b/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600467    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212431094s: [/var/lib/containers/storage/overlay/84e2978c7bf6f8a82e9cb045dcd37b94011845204a16c93bb3912ff1c3d5582e/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600491    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212453768s: [/var/lib/containers/storage/overlay/ced9b4ee10b95de9e21ed7fe4450d4c9fcc97bb9e1d3ebda22fd9d469e5f4292/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600516    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212477439s: [/var/lib/containers/storage/overlay/9dcf2fbc8f2a12b867ec625655279863edbfabf7bb03acffd8482f4d3a0eae26/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600541    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212500897s: [/var/lib/containers/storage/overlay/1b20b5f4508512ef1b3778f562c5f97da26da6bf4d0856c4ef9a07f44819b3f0/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600566    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212525479s: [/var/lib/containers/storage/overlay/4756e8c484bb99f308a787d9771c513734c834c31e16597655b029fb5dc0d243/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600593    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212550601s: [/var/lib/containers/storage/overlay/7f098c561ddc879a9311701be103276e1d03f82ab3efa34f2b873a3e424f0d21/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600621    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.21257807s: [/var/lib/containers/storage/overlay/da83a0515dd606978929d985eda34e4783473196836c68c43bf562fe9466bd7b/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600647    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212602977s: [/var/lib/containers/storage/overlay/4aff5b253dee34e4a614e7a9c0214a5c12d6acd592dd21d04cb698f2042a937c/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600672    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212627432s: [/var/lib/containers/storage/overlay/6304d0fceeb709e85ca572090857d08706f669cb1c3222985b10ace2393d51cf/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600698    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212651682s: [/var/lib/containers/storage/overlay/959687ac2acb5f81ac7bb8a093e2668af6b0018b8cabcc37179678695dd639a3/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600723    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.2126755s: [/var/lib/containers/storage/overlay/1c9f6f61c3280851db3a1425f487b201d7a593a4e27a560538129810b3169adc/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600748    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212700303s: [/var/lib/containers/storage/overlay/b428ce3434de02ee9be0b02f069e7c0e10cfc97ead5bb22bcc622bbc6836f7d3/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297941 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600775    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212726202s: [/var/lib/containers/storage/overlay/e64870204381c7a7f2555e3b5fe92951842ced33934d9f090c927aa4bb74212e/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297941 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600799    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212748818s: [/var/lib/containers/storage/overlay/a1ee19348961dae79c087e8ac3bf9e59ed7a33f37091eafee1f692eee68bc0a0/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297941 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600838    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212786755s: [/var/lib/containers/storage/overlay/ec89134dec345974daece69f9b8dd6137e843685bffddf149e15d295f298817e/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297941 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600864    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212811441s: [/var/lib/containers/storage/overlay/b85fed92aa7942c2d1b0a3ebd90904062437ad83478c28e6d785342de059e340/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297941 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600889    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212835297s: [/var/lib/containers/storage/overlay/8099f23161d5eec2e26bd9fd0d29e0723c872c3a2553491cef9562905ddcb497/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297941 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600916    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212860823s: [/var/lib/containers/storage/overlay/25756dda2621604e75c6eea74f5fb75349668aa5a63ad77893fd3405a7c6f263/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.297941 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600943    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212887132s: [/var/lib/containers/storage/overlay/f43459bfc40568911db1c28d46def37ad81ebdd789dcbc7aef804efa5574b108/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.298226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.600967    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.212910645s: [/var/lib/containers/storage/overlay/7530ca161075d72465e087d6cfea7fe9079bd67c737753ce9107f7d5bf04700b/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.298226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.602105    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.214034023s: [/var/lib/containers/storage/overlay/1c6d416b95253d64cbccfa7f2605c9d443f62eefa0f5d0626161a5261456fac0/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.298226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.602131    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.214060557s: [/var/lib/containers/storage/overlay/b193bcf543ea8163b7da56ebffe1a592dd1c275d976dd60b8d19c46f9f897138/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.298226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.602155    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.213919422s: [/var/lib/containers/storage/overlay/4835e8154ff3f70d907d90cedc5ad8524fba4d2106f30253d83cdcf05ba99c46/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.298226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.602179    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.21394252s: [/var/lib/containers/storage/overlay/d3bb1d8cdab5766b606d96bd1ebc61c634339df13a703533522356720f1907ac/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.298226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.602205    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.213966956s: [/var/lib/containers/storage/overlay/c3f17c66c7e230b3ba16f42aafdd369608de60e44df877a9cb942d9ed4c41247/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.298226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.602231    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.213992094s: [/var/lib/containers/storage/overlay/5517713a95fc08e0575503ff1195bf3084c018054f1947590ab93829560900bd/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.298514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.635679    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" probeResult="failure" output="Get \"https://10.134.0.47:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.298514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.635870    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.201:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.298514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.635918    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.298514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.638552    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.187:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.298514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.639244    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.173:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.298725 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.653182    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.176:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.298725 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.659300    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" podUID="d837dde2-3939-4400-8ea7-fee58c162983" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.116:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.298725 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.659379    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j" podUID="e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.44:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.298725 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.659436    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.200:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.298725 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:00.664901    4456 logging.go:55] [core] [Channel #6618 SubChannel #6670]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.187:50052", ServerName: "10.134.0.187:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.187:50052: operation was canceled"
Nov 02 18:11:11.298997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.664967    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.298997 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:00.667902    4456 logging.go:55] [core] [Channel #6619 SubChannel #6671]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.173:50052", ServerName: "10.134.0.173:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "transport: Error while dialing: dial tcp 10.134.0.173:50052: operation was canceled"
Nov 02 18:11:11.298997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670089    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 37.865656035s: [/var/lib/containers/storage/overlay/c7562bd346d831342663c585fc20338a2b2e898c53786e7fb087a0324f4f7e13/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.298997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670319    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 37.865877985s: [/var/lib/containers/storage/overlay/0234c330ad731526e0f7c2e86fd1692f4cb5053f776e389d790b7710b7d2e028/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.298997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670363    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133105855s: [/var/lib/containers/storage/overlay/fe9920e5a7065775346550e5358cc11abb4093be2682b270b828978d1fc3bdbb/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.298997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670393    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133133007s: [/var/lib/containers/storage/overlay/02ed3aed9d7d1f9932504189ec7d2043ab6ef5260de12ca49e1cf4bcedd555cf/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670426    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.13316451s: [/var/lib/containers/storage/overlay/dd180952a24151210f392620d65614521677a90358a32e8dcac6216703ef8c5e/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.299243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670456    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133192688s: [/var/lib/containers/storage/overlay/f9027342e93f93cf2207e54c54a505f7bd2ec4fc18d0fb5e35c550a0d937439e/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670485    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133219374s: [/var/lib/containers/storage/overlay/12a0c2eeda822e1ff661410a43076cdd6aedef1ccef4240d7456685f94696af9/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670515    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133248581s: [/var/lib/containers/storage/overlay/47c6f9fa4e81d5b6bb6cb4211eff0df7ee33a694a7d6141f384651326b031a13/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670545    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133275564s: [/var/lib/containers/storage/overlay/eea65d58e8e5145ae3222bd58cd763318191e7e1e8a3911be816c6f629e9177c/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670573    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133303094s: [/var/lib/containers/storage/overlay/4e361aab9892c59ae1c857a4c3f26675ab711d42dbb2eea22c0038a24983687a/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670602    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133329713s: [/var/lib/containers/storage/overlay/88bc22348df8e280411872f066dbf14feb3968b22f192d53027960ffae02080e/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299532 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670631    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.13335773s: [/var/lib/containers/storage/overlay/a7f26589ec75d499102c1e32470fe7d9c2e016680764619bc4f468b50424ee5f/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299532 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670659    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133384099s: [/var/lib/containers/storage/overlay/7cd1c406d79080a5de52020433aba1bd1e3439731a6008c91a44ba0d734e0168/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299532 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670689    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133412737s: [/var/lib/containers/storage/overlay/7a274702c3ffdc2535cd43c3814dd50ea0416d2fecf6b613833a80a6cd3be072/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299532 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670719    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133441472s: [/var/lib/containers/storage/overlay/889a1e50cd346c2ae6c3699e22fc6333114a9fef99ca78329cab0059d0af21f8/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299532 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670747    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133466032s: [/var/lib/containers/storage/overlay/12034bda1ff91295c47a5affae545ed5da549ac45cf44e53b8bd75905db1b189/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299532 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670781    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133497561s: [/var/lib/containers/storage/overlay/e4f11f93cbfd2fc79389aec87c75cc993913f4c053e02fc2bef087f73afd5d7d/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299532 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670810    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133525543s: [/var/lib/containers/storage/overlay/510d5b5021012ada853fd2e3b25638149887bac7a79939d0bf56c0262cc66081/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670857    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133571004s: [/var/lib/containers/storage/overlay/f1bf059e369a60e8c8c66e453cf0724779ec1592faffe0ceeaccedee71095804/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670889    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133601504s: [/var/lib/containers/storage/overlay/578d0af976732c1c9e43c84eae06aa0aa97b984f5931bc98ceceb74deb7f87b2/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670917    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133627779s: [/var/lib/containers/storage/overlay/8da893366900c89f1a49f0a89cff3a847547158c42b584792c9ed31f41e77cc2/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.670949    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133656624s: [/var/lib/containers/storage/overlay/bd48af5b0a05e1435415768044430abaf131384d551707641cd919d54da040a4/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.671173    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.133875676s: [/var/lib/containers/storage/overlay/70b4ec9c282135b6aad8758fa5f78d709532a56a100a349d35d742572e1db126/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.299830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.671185    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 37.866744294s: [/var/lib/containers/storage/overlay/f41fd51d26fbd21de386cefe58613bae1900970a633ac15435e1675fb963123a/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.299830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.671205    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.132882682s: [/var/lib/containers/storage/overlay/e7f6af957917b0de1eac2c0f835eac9f4d66364bc548a8587261fc6bec5607da/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.300119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.671237    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.132910429s: [/var/lib/containers/storage/overlay/b4590eecb2fb63bb3d6ae41839b6301350c179ba2a20691909813f55406e0826/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.300119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.671267    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.132938974s: [/var/lib/containers/storage/overlay/4ab2f93d76b331d264037ae17bd9910369889be014d74845e96039454c48bbf7/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.300119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.671295    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.132966701s: [/var/lib/containers/storage/overlay/1f6300bdb8f1111be5619f5fe0108ffbf0081e9e27bd7de967b906b8b3ad33d1/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.300119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.671325    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.13299383s: [/var/lib/containers/storage/overlay/76d53787e52a6a6617da848277a740036843744ce34ec997ac569eda8823e979/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.300119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.674009    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.135681086s: [/var/lib/containers/storage/overlay/c54941b7bb0f708eefe849545e48082649e74c106518b009fddf3e0544c6bc0f/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.300119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.674404    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.432035685s: [/var/lib/containers/storage/overlay/aff244db02eacd0334ea3e804c3284703be41cda08f02e8f8d0a2997784e5b04/diff ]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.300119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.674450    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.962179831s: [/var/lib/containers/storage/overlay/2e0adf259b4f46a31705cecfb5e7924b04dcec1097b7bec6afcbc60c92bfa8a4/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.300405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.674484    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.962210608s: [/var/lib/containers/storage/overlay/cb2f4d21c7e74b313922bb09d6b5f3805778f08b0046d9b0e26544665af451da/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.300405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.674517    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.96224148s: [/var/lib/containers/storage/overlay/fab27cad89ad4b3e8d0b5c4ac0b7b3a7912aa4fa9886a06d4abd5720cb2eb720/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.300405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.674550    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.962272228s: [/var/lib/containers/storage/overlay/76e05be6c6727d8ded946e030da1c274a89ca48d16a9caf69ad1842aaef2e487/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.300405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.674582    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.96230283s: [/var/lib/containers/storage/overlay/8db47c776480e39a5f679a3dfbd86e84aed657e023ade55f978a8f68f01ea9f1/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.300405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.674616    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.962334646s: [/var/lib/containers/storage/overlay/48451e9990851b8e024d8b3972a86ef0a0c9f6439dec07d6b3ef16f8fae2c355/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.300405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.674650    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.962367291s: [/var/lib/containers/storage/overlay/08a91d46928182b4e183ee7ace17ce789177b554acc520da9618f6d1c993924e/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.300405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.674683    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.962397833s: [/var/lib/containers/storage/overlay/2269cfd2edca6d6508ccccdbd523a9546a86e6f9de9c2f183ed3f2e8f50fb2fa/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.300684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.674716    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.962430293s: [/var/lib/containers/storage/overlay/655a7e468823c9c970918a567d0b0ed61e5e4f40137739e6fa8ac19b456819a6/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.300684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.684924    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.170:50052: i/o timeout"
Nov 02 18:11:11.300684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.685864    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.300684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.685898    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.223:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.300684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.686412    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" podUID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" containerName="payload-processing" probeResult="failure" output="dial tcp 10.134.0.172:50052: i/o timeout"
Nov 02 18:11:11.300684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.694875    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.300945 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.694912    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.300945 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.694977    4456 patch_prober.go:29] interesting pod/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.300945 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.694999    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.174:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.300945 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.696196    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.300945 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.698976    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerName="payload-pre-processing" probeResult="failure" output="dial tcp 10.134.0.175:50052: i/o timeout"
Nov 02 18:11:11.301156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.700376    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-hr8vz" podUID="e73be583-2a26-4379-8169-0a6b8784a11b" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.134.0.26:9403/livez\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.301156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.701519    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.301156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.701575    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.301156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.702414    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5knvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.301156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.702470    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx" podUID="8ef79a10-292b-42d9-aab0-53bee3e43ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.301369 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703003    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.301369 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703039    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.301369 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703237    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.301369 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703269    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.198:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.301369 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703347    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.301574 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703381    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.301574 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703582    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.461209004s: [/var/lib/containers/storage/overlay/9f3bb7fa13dcbf6c99ec343710a35437e7f61d66d1bdac63538a05c396354291/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.301574 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703622    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.991332155s: [/var/lib/containers/storage/overlay/4e67e8cd7fb897bc5116e504bba9a29a8ed30c4f9f1fac62d47ec5892d590847/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.301574 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703652    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.991359577s: [/var/lib/containers/storage/overlay/0024ef56118e59543f04eba902323a6c84cea8671f97bb98441651fafd3f1fb9/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.301574 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703681    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.991387923s: [/var/lib/containers/storage/overlay/a18f6543a5970d2a5877e9cb019fa68742596c240618559891255c531214ebc1/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.301574 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703710    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.991415076s: [/var/lib/containers/storage/overlay/18864169905dab44c7b7c46db110c9b2e1d7a5b103538ab226d089b5c195ecec/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.301847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703739    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.991442832s: [/var/lib/containers/storage/overlay/c155031b458209f5c3cd2b2c6dbb1583f454b4f2ea510626f12b6fd5ab84995e/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.301847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703775    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.961376776s: [/var/lib/containers/storage/overlay/c7363f8222650c51f900a246744d12fb88176e629d384d106cefbae7c47dd7cd/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.301847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703844    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.961449135s: [/var/lib/containers/storage/overlay/2a7f5aa5ba39a921bd515553a1089c5e8bb4a586c47eda778d4017ed30ecc198/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.301847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703876    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.961479862s: [/var/lib/containers/storage/overlay/7049a5acdc2384d17d12da111ac7b7e2c0312fd1c17bd071788f8bf2808c768a/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.301847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703905    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.961506814s: [/var/lib/containers/storage/overlay/95d0719995519e96ca4e2ef099d1cede945650fd1a2d8d4c03c7de64145ce3de/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.301847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703935    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.961535241s: [/var/lib/containers/storage/overlay/e63da92cc01fe6e396c33175c254611578c02ed94dc2baa9164216cbb22bf034/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.301847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703964    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.961562284s: [/var/lib/containers/storage/overlay/1c9abd01101f122b64b87bfa59945cc9d609da8965b554240c8857b6517c873e/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.302140 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.703991    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.961588741s: [/var/lib/containers/storage/overlay/071d5a818923690326943183a456d5f42cc7cf06a45bbf46dce225490af0ccde/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.302140 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.704021    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.961615347s: [/var/lib/containers/storage/overlay/8a34801be51a556958c8d17b00f8e1380149782e0320cbe6b256572a3ef0e18e/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.302140 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.704320    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.461923143s: [/var/lib/containers/storage/overlay/8a61e7e207706ded21f782a468d9622ae8cfb244d32ad9c9d3f18363aca5ad52/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.302140 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.704587    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.462214306s: [/var/lib/containers/storage/overlay/229a9ce45461e83bcf894d5fdec3589a18c0dde28fd30e6c2f7267725d721843/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.302140 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.704867    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.462491989s: [/var/lib/containers/storage/overlay/46c87d9588b5c4153befd46bcce313bb8d9eed33c45f7c4b76fbab7d7567da57/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.302140 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.706610    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.302381 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.706647    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.195:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.302381 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.706705    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5bqck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.302381 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.706723    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.37:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.302381 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.706768    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.302381 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.706783    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.228:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.302582 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:00.706895    4456 logging.go:55] [core] [Channel #6634 SubChannel #6637]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.116:50052", ServerName: "10.134.0.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:40462->10.134.0.116:50052: use of closed network connection"
Nov 02 18:11:11.302582 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.707885    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.465508814s: [/var/lib/containers/storage/overlay/0a1b5af1d2fbdef3ff1b7294208f7f4115ccab6c1c6f56a4dbd94ef547fdbb15/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.302582 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.707926    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.848255843s: [/var/lib/containers/storage/overlay/37d56634756aeeeca9171a043730fdb311d7d6352312d621fb8e69abd123ca4d/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.302582 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.707957    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.848284451s: [/var/lib/containers/storage/overlay/9a65488a0d46fd5c20f43cfe794eb9f7883b34fba7149707cfc381b54e62c6dd/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.302582 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.707987    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.848314648s: [/var/lib/containers/storage/overlay/4a89652208b5a1fd16b1a02605f67a9b20d35811800ea0052f9e3a8ea729ac7c/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.302582 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.708015    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.848342413s: [/var/lib/containers/storage/overlay/39e7e1a3433145ba62e41a80782f5b1091cd38d1872bf5a708709525a34f7ff7/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.302843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.709034    4456 patch_prober.go:29] interesting pod/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.302843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.709174    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.217:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.302843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.720261    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" podUID="e141d845-db9d-4f94-b05e-cfb7848e8fef" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.180:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.302843 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:00.720494    4456 logging.go:55] [core] [Channel #6621 SubChannel #6676]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.171:50052", ServerName: "10.134.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:45494->10.134.0.171:50052: use of closed network connection"
Nov 02 18:11:11.302843 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:00.720914    4456 logging.go:55] [core] [Channel #6629 SubChannel #6630]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.201:50052", ServerName: "10.134.0.201:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:33818->10.134.0.201:50052: use of closed network connection"
Nov 02 18:11:11.303050 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:00.720973    4456 logging.go:55] [core] [Channel #6635 SubChannel #6657]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.44:50052", ServerName: "10.134.0.44:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:55000->10.134.0.44:50052: use of closed network connection"
Nov 02 18:11:11.303050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.721001    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.303050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.721027    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.226:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.303050 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:00.721104    4456 logging.go:55] [core] [Channel #6638 SubChannel #6639]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.200:50052", ServerName: "10.134.0.200:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:58136->10.134.0.200:50052: use of closed network connection"
Nov 02 18:11:11.303050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.721128    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.303257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.721147    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.196:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.303257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.721246    4456 patch_prober.go:29] interesting pod/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.303257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.721281    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.126:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.303257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.721246    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.43:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.303257 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:00.721771    4456 logging.go:55] [core] [Channel #6646 SubChannel #6647]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.176:50052", ServerName: "10.134.0.176:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:49842->10.134.0.176:50052: use of closed network connection"
Nov 02 18:11:11.303460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.723386    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.303460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.723435    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.40:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.303460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.731713    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.872033175s: [/var/lib/containers/storage/overlay/78f0ff08138ee83d62f74d0f093a65703c9dba5248bba14dc94fce7c62d2eb68/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.303460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.746992    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.175:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.303460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.779697    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.375175389s: [/var/lib/containers/storage/overlay/5c9ea955e524e35aff9b30ed78d97e370413ede7d4597334da2209e4500672c6/diff /var/log/pods/openshift-ingress_e2e-inf-a-872354-openshift-default-59df9df7-vnmfl_3c8d9150-66d1-4be8-8bca-3782fb0eac36/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.303660 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.779745    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.372650723s: [/var/lib/containers/storage/overlay/d6bd7fb5658266e0883715f2708154a35f91862981569c388e3cb5f866afeeb3/diff /var/log/pods/openshift-ingress_e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7_f56cfa88-cc79-4d08-9111-b47d940533ba/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.303660 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.779772    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.37437247s: [/var/lib/containers/storage/overlay/05b2f9768ded3bb89850b28832c2c3463a2ffa7b9eab99ff39d4e91999d29770/diff /var/log/pods/openshift-ingress_istiod-openshift-gateway-7ddd9cfdc8-tdx82_0f2089f4-8ec7-45ce-a2df-0bf024b4158f/discovery/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.303660 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.779793    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.372233076s: [/var/lib/containers/storage/overlay/61dba17c91911e3fe21e3760810cb0fdaa44600557f817d2a6d5f9e5af198421/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn_3d96e066-10c0-4fe2-9e24-ffd6195b22af/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.303660 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.779838    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.375078042s: [/var/lib/containers/storage/overlay/7840ca7106125a4b22fa575aa093373b243546e0cef5c452d1dd3a71532c03ea/diff /var/log/pods/openshift-ingress_e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt_856c2dca-bad2-4f62-a536-1ca0e09f0a12/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.303860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.779861    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.373999037s: [/var/lib/containers/storage/overlay/a55dca0d0429c95641f315985403a890eb29899befd46ff107f67affa1d6343d/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw_5b0ff069-2a17-489e-ab3d-6d5461bffbf4/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.303860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.779883    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.372029015s: [/var/lib/containers/storage/overlay/1b551b04f397d2a17bb4b72d2a9e13ea7b255a0c2887bd5f17c4e17a3c5d3890/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/kube-rbac-proxy-node/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.303860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.779905    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.372050099s: [/var/lib/containers/storage/overlay/39036cdcb7e7029ac2d15241aa50d868ff1a012fe62623885c01ed6f7528dbce/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.303860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.779927    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.372070378s: [/var/lib/containers/storage/overlay/afcf005d130d4b0994316011111d3ba89bede3db3572360e1b8cddd15c5a4284/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304025 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780003    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.373393162s: [/var/lib/containers/storage/overlay/1488f5a27e0831e0c877985c837f753313a612723997edaa573902b65e840bd5/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f_5ce3335c-36e6-4018-a264-5b79085f09f3/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304025 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780028    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.375758963s: [/var/lib/containers/storage/overlay/8fe16763de809dbbc885c5aa483909cdeb3f142a735700e1976d0912af066878/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2_49238ac8-57fa-412a-88bd-681364a0b455/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304025 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780052    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.374980257s: [/var/lib/containers/storage/overlay/9437d7121cf0584354b918af2df36d1e6733adeb8e3ad8e653ee51e2eaf2b268/diff /var/log/pods/e2e-models-e2e-worker-w1-120394_premium-w1-120394-kserve-669cf87f88-rmj4s_5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34/main/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304025 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780080    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.372217555s: [/var/lib/containers/storage/overlay/b7702443b08031bf70dfb72c9da1f832917de0a24b9b53011dbe509513501266/diff /var/log/pods/kube-system_global-pull-secret-syncer-mmhd7_8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d/global-pull-secret-syncer/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780100    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.372236863s: [/var/lib/containers/storage/overlay/44d9303590300b2067d19221719afe781447e01e7a3698b47e26576b04692428/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/config-reloader/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780133    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.372256783s: [/var/lib/containers/storage/overlay/37c279f7138b64cf848e046dd0159412d2f60a6e3f561819a069d84c7835dd5e/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/ovnkube-controller/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780157    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.3722916s: [/var/lib/containers/storage/overlay/436b55b989bc697feca6cac52a8bbeabf4607101e8e2d4cf93eca0342b9c313b/diff /var/log/pods/openshift-monitoring_node-exporter-r45c5_8c89d394-c878-4069-b3a9-cd0abc822fec/node-exporter/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780178    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.372311077s: [/var/lib/containers/storage/overlay/349bb891311b428d7245edc3a45686b0ebfaa1beb5ddb02e1c5186f29d1a0a38/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/nbdb/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780199    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.372330373s: [/var/lib/containers/storage/overlay/30c6ef23d54157e2314449730cec7e09831a7cd2d10ec7e7cdc43a4ced04dd2a/diff /var/log/pods/openshift-monitoring_node-exporter-r45c5_8c89d394-c878-4069-b3a9-cd0abc822fec/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304391 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780220    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.374987177s: [/var/lib/containers/storage/overlay/2451471e892c455bc0429337eed2450e847bf657dfb5b21286782c82632452ed/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz_b5d45c38-1cef-4aa9-a519-98aed00e8727/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304391 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780241    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.372369608s: [/var/lib/containers/storage/overlay/4e57f3a2be8d4ab9b3c25af8d27d43230ec513dda8164ae6c781b961a859b92b/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/prometheus/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304391 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780262    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.372389084s: [/var/lib/containers/storage/overlay/a6efa294561d8867fca81b5aa12f5b9f4ce0955f3122e0d2c68924d7a6db53f8/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy-rules/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304391 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780283    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.372408529s: [/var/lib/containers/storage/overlay/ec5326cd4c57e5289c85be1ed540648b29113e91e4b3f4a2026c4c66a8e87d65/diff /var/log/pods/openshift-ingress-canary_ingress-canary-n4r8t_ba5cbec9-7649-462a-b110-11ebcc5d6d78/serve-healthcheck-canary/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304557 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780302    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.37242696s: [/var/lib/containers/storage/overlay/48d5c82df7e650a089ea0fa91ed0bd4c1b6a9b1f380d8d8db83882fc5cc1bf60/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-driver/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304557 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780322    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.376244068s: [/var/lib/containers/storage/overlay/9de66e4fccd5433ca81c75c6841035983055f1565dac16f5b02b99f158a7e535/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5_3d398f17-735a-4d4b-b1cc-5b24fd266535/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304557 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780341    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.372462404s: [/var/lib/containers/storage/overlay/ff286283e890b2f02d7b35cbc69ffb7a37f1ddbe5461c9aabad15b3d6f880cd6/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/kube-rbac-proxy-thanos/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304557 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780360    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.372640759s: [/var/lib/containers/storage/overlay/97e54822e3779b90c1439e95679938651a2f36f363a4df8db562bacd72f19b57/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2_97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780379    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.374024454s: [/var/lib/containers/storage/overlay/98ff010f77e7d96f5843be7a3a27066c1c197ffa6885b42fa9f473d20af37b2f/diff /var/log/pods/openshift-ingress_e2e-inf-a-872354-openshift-default-59df9df7-v5qhw_32802945-5e87-4339-a7eb-33212aab7ba5/istio-proxy/1.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780399    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.374278717s: [/var/lib/containers/storage/overlay/fdf2869abb29b990967a0fcf0bb0ef8a663dfc34c944a5841012b183fb830979/diff /var/log/pods/openshift-ingress_e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4_92ad0220-435f-4ae8-b743-3421ce98e6fb/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.780421    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.37253663s: [/var/lib/containers/storage/overlay/6adcc40b109806b8d4dee9b3334b845a3feca9388fee349574a27397b371bd0e/diff /var/log/pods/openshift-jobset-operator_jobset-controller-manager-77cfb5f94-gfbr9_6957a218-62fc-4dd8-892e-4477b08c046f/manager/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.304724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.795272    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.170:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.304971 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:00.795542    4456 logging.go:55] [core] [Channel #6651 SubChannel #6652]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.180:50052", ServerName: "10.134.0.180:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:36906->10.134.0.180:50052: use of closed network connection"
Nov 02 18:11:11.304971 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:00.796959    4456 logging.go:55] [core] [Channel #6654 SubChannel #6655]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.175:50052", ServerName: "10.134.0.175:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:33092->10.134.0.175:50052: use of closed network connection"
Nov 02 18:11:11.304971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.813912    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.571532566s: [/var/lib/containers/storage/overlay/b47b9dd6c514448c5e5a1ae04c109b9960c3981afb61d47419b7ec217051ba27/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.304971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814001    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.427782553s: [/var/lib/containers/storage/overlay/b20116e8eeb7eef067efbcb5dd7bded5ed2f9d4621f36c8e5f282b97c52fd109/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v_6ebd3467-841a-4331-b149-8acf6e60c12c/maas-api/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.304971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814034    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m5.979136201s: [/var/lib/containers/storage/overlay/e68ebf6860926a21a0cbfecd410ae12a4c05fb62dae7bcdbf219e8a050cbd79d/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8_e4c1132f-05b5-4664-84a2-ece57727b12a/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814068    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.407285936s: [/var/lib/containers/storage/overlay/1d679dc376e8b4b61ee5b35c347f887a08cc74c2c68502794ffcfd9744b157de/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq_5e54b0f0-5c3c-4441-a2a6-b452b9715e2e/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814090    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.406239064s: [/var/lib/containers/storage/overlay/0d74f9f4fbbf600087eeb718bd1494b6cee4f94be2e06fed79e7cc0e32173cf4/diff /var/log/pods/openshift-dns_dns-default-zmrdt_f59eba5d-a500-462e-9ecd-4ef9e553f873/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814147    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.406230735s: [/var/lib/containers/storage/overlay/38f6c3a3bf1761682b345c03cb02c31f484b894296aec5d499ce5ae2e5601af5/diff /var/log/pods/openshift-machine-config-operator_kube-rbac-proxy-crio-ip-10-0-141-87.ec2.internal_375359043409c490a8ed0d21b8afeaf5/kube-rbac-proxy-crio/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814173    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.406320427s: [/var/lib/containers/storage/overlay/d0014aabc7632aacf8f230d1d606750084728ea64d88ba0a6c129361bdd715f8/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/thanos-sidecar/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305403 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814195    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.429407379s: [/var/lib/containers/storage/overlay/42a7abc9d31e7e91afb1dc9f586b5fc298c1107b48f881228866c2735a90fee7/diff /var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-q2mbj_cae854c8-88f7-4201-b5d7-4e7c59fdf833/prometheus-operator-admission-webhook/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.305403 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814224    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.429433762s: [/var/lib/containers/storage/overlay/1cfef9a8ec66cc7e72a2cc8f5db5f517a05984ada8feb9b6fcab0c85033fb6e5/diff /var/log/pods/openshift-multus_network-metrics-daemon-tk8ht_dd733778-7002-4a26-a4e8-4525d8f6bee9/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305403 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814247    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.408664142s: [/var/lib/containers/storage/overlay/35ea380649cc6d0f8dc0e907ecb9ef66961662f67283708985d685f75f06497f/diff /var/log/pods/openshift-ingress_e2e-worker-w1-120394-openshift-default-6459698f85-j24dq_acd44782-84e1-411a-9a1c-6d21456733bb/istio-proxy/1.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305403 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814266    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.406943348s: [/var/lib/containers/storage/overlay/83f8cdd611cedca899243d5a228c934d7754b14823c1219db9a0f8ced5da014e/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-5bqck_280a3e6c-c540-4c66-b963-c678ddca6e40/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814333    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.406475259s: [/var/lib/containers/storage/overlay/903a7a7c7002a9034e9b89db1ce800ddd9a85962e7522211d8fa957ab69f9ec0/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-m89vg_9257d59b-6aaf-423e-b7ed-2e64dbdfa535/kube-rbac-proxy-main/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814357    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.429558151s: [/var/lib/containers/storage/overlay/87c11e77a69c1256c563023edeadf4b64a91805f1340898de6ce44397e54a787/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/kube-rbac-proxy-ovn-metrics/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814390    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.429590545s: [/var/lib/containers/storage/overlay/932ab5e5702f5c7c79c2d32910577875654df1477eac0759bb03b85f11ce0d04/diff /var/log/pods/openshift-kube-storage-version-migrator_migrator-66df5d496b-lxxf2_8064da0c-fc5d-493c-bcdf-4327716d4cc2/migrator/0.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.305567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814411    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.406552679s: [/var/lib/containers/storage/overlay/68adcccdea3eba205d654c160d713ccd9fe3f136335e3c8c4fd9800b1d4da9c1/diff /var/log/pods/openshift-network-diagnostics_network-check-target-nttxj_a974c7e3-bf97-4575-bec1-c09fcd018f0d/network-check-target-container/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305730 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814433    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.406572638s: [/var/lib/containers/storage/overlay/7c41892d8c349d1ad9ae2bbe72fa8e68344bac8db842e0614212577491c66d1d/diff /var/log/pods/openshift-monitoring_telemeter-client-79c994cf5c-ckj5h_e065a5c3-95fd-4ac8-a287-8e1d20219201/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305730 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814514    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.406643735s: [/var/lib/containers/storage/overlay/411d81d088de93b0674ea4f2dd71b25e85f37341baf36fa8546cc1221ec492ad/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy-web/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305730 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814538    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.406645312s: [/var/lib/containers/storage/overlay/1d99eb2b6b4423a5208899af1d81194c5531b55c6df383327a20817e35f00610/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89_8c60dfaa-dabe-4726-91b5-165443ff3971/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305730 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814569    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.429762292s: [/var/lib/containers/storage/overlay/2850c96a313d433f759e56135d890d3c3b9d4452f3e507896aa256d912e71270/diff /var/log/pods/openshift-cluster-node-tuning-operator_tuned-n9fk9_f4b72e7f-cc4c-4662-a923-c184069aa118/tuned/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305911 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814592    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.429783571s: [/var/lib/containers/storage/overlay/69e9e5b9009b44f81cb13f7e89161acc514055b4129ec8240b07d4f06c225a45/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-node-driver-registrar/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305911 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814651    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.429841841s: [/var/lib/containers/storage/overlay/065839dc8c2c6a550d10b437944e397361cfc89bf9ccb63c18a6d705e7669644/diff /var/log/pods/openshift-multus_multus-7qzkz_f26fab1e-e6d4-4caa-bef6-81a09d78c0de/kube-multus/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305911 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814673    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.429863348s: [/var/lib/containers/storage/overlay/833351d2ebd14fa986d0203e8af3254003326af193a6655f85b7084ad760a6ec/diff /var/log/pods/openshift-ingress_e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf_999df778-cd9b-4b41-852d-4661a7326483/istio-proxy/1.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.305911 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814705    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.406821325s: [/var/lib/containers/storage/overlay/cf8db5461455b84412026ed2fb5a9f7f27755e4db156859b09da808cfbc028e3/diff /var/log/pods/e2e-models-e2e-worker-w5-863b87_distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f_bffc7364-3157-46e6-8fba-616f128b60cf/main/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814726    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.406839491s: [/var/lib/containers/storage/overlay/da5b82203a9243942a4be5d58ab45616354c2ad53c09072b04c350a996fc6bd7/diff /var/log/pods/openshift-ingress_e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s_5bb882f6-fd8f-46ae-9396-b52498c5d9db/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814746    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.429916284s: [/var/lib/containers/storage/overlay/6da4544d5bf904910dc0072ef528e9149345d962762f71ecb9b0ce270fe91dc2/diff /var/log/pods/openshift-insights_insights-runtime-extractor-6bfcf_da9f1b66-0e1b-4c7e-a947-a897c00ccc19/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814802    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.429968734s: [/var/lib/containers/storage/overlay/089a992d10746017d3c97ab20d1ddf9b955c8babb89ca5a1613bd5a1985f773b/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-m89vg_9257d59b-6aaf-423e-b7ed-2e64dbdfa535/openshift-state-metrics/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814882    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.407032681s: [/var/lib/containers/storage/overlay/a2aa71b482739c7b0a0cc470b85d5d8947410f51100d4866f063fe123ede7d3e/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr_22b556c1-112b-49d4-932c-0143e912a54e/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814906    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430072086s: [/var/lib/containers/storage/overlay/6938a4b799445ead8836745535273347f3c908c027c27cae3317fe23a343f2cc/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/northd/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814967    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430130549s: [/var/lib/containers/storage/overlay/38cc698afa3d0a22ee2e64e3fa8b08e084747b51f63934648a8d8162d387223b/diff /var/log/pods/openshift-insights_insights-runtime-extractor-6bfcf_da9f1b66-0e1b-4c7e-a947-a897c00ccc19/exporter/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.814993    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.407110581s: [/var/lib/containers/storage/overlay/42bab2822c3e5230896f9c86b84693de76f186c67813e61eb7792866accc56b3/diff /var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4_6e44698a-838e-4eea-b956-585673963ed9/main/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815014    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.407137203s: [/var/lib/containers/storage/overlay/1a5df3100db9dd3cfd75b7f681887de0a6a1fe77ee9e7827781cfdc3b41a2d4d/diff /var/log/pods/openshift-ingress_e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck_694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815046    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.40720193s: [/var/lib/containers/storage/overlay/3e001268063ff1680111524029f98c7759f3a4feccf0afcf7064a2ca68c37a83/diff /var/log/pods/e2e-models-e2e-worker-w0-069fa5_premium-w0-069fa5-kserve-699ddf764c-xmspr_9e85cf3f-eb7a-452b-b716-26df5b391065/main/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815067    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.40717996s: [/var/lib/containers/storage/overlay/f4bf622192145887ae6ee7c0202271dd59950c44754a850ccabfcb18cbe5c0a3/diff /var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-m89vg_9257d59b-6aaf-423e-b7ed-2e64dbdfa535/kube-rbac-proxy-self/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815124    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.407240903s: [/var/lib/containers/storage/overlay/57432df54ab94c322fcc3f8ed7e5d69537ef205267b7e8198fece234918f9d44/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n_85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815151    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.407261892s: [/var/lib/containers/storage/overlay/a568d4ed72e2a436fe819cab66f62bd95c9b47563cb98179484da40963eb831c/diff /var/log/pods/e2e-models-e2e-worker-w3-5eedab_premium-w3-5eedab-kserve-786db78999-f8lbq_f563607b-3be4-4194-aa10-81e8bf2cbf6e/main/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306561 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815172    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.407281014s: [/var/lib/containers/storage/overlay/3c75f5f1fa0b98e778e718536ee383cfcc5ca413d1084810926fa3df6d132250/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5_02a2dbe6-baeb-40c4-be0f-835239be331f/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306561 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815207    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430356851s: [/var/lib/containers/storage/overlay/c032743b3f5164f05268a34d27f7a38a2df55f102c647c344a128377fc34aefd/diff /var/log/pods/openshift-ingress_e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx_81fd82a8-2bbf-43c8-962c-f8384287dc1f/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306561 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815230    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430437188s: [/var/lib/containers/storage/overlay/acc66a71ede97842cda782e51e0adbea72945cd7fa0b1af2fc4160e227b7062f/diff /var/log/pods/openshift-ingress_e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5_0996b2fa-40e8-469c-9ace-75458691fa25/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306561 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815288    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430474342s: [/var/lib/containers/storage/overlay/6890c8f8933a1e87b2636594da397d70dd586fa37081a11d1320e3554116bda7/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-vrvf2_07f874bb-122b-4794-ab59-736495e5cfdc/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815312    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430475008s: [/var/lib/containers/storage/overlay/7c82bb52ad040de109f8f96e4fde961296a9a03788caea2ee7088d4b4b9b113b/diff /var/log/pods/openshift-ingress_e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt_82ba53b4-e91a-4299-9498-f5abcd34576b/istio-proxy/1.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815332    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430477839s: [/var/lib/containers/storage/overlay/861acfcf2321b99e26e85c3154bd9202cc22f4d667e5041f027bd7d24464ecb9/diff /var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-liveness-probe/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815362    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430506855s: [/var/lib/containers/storage/overlay/48b947c743d65c0ae34868fbb47aeb890dde93159790d378edd81be829b75cfb/diff /var/log/pods/openshift-dns_node-resolver-hhsrb_ce8b7505-62a5-42f1-b432-d77200c21d2d/dns-node-resolver/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815384    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430525884s: [/var/lib/containers/storage/overlay/7543a86d2f7dc71bebf4306a35f19682692a0213b5009a3a8a2a0250a1d74670/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/thanos-query/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815404    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.43054573s: [/var/lib/containers/storage/overlay/1550c3e2e2a45dcfd84afdd33aa3690aa5dcb9b9475cb07cc27b0e0b34c41bb3/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/sbdb/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815463    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430603431s: [/var/lib/containers/storage/overlay/e34d2a2f5077d6161aee9054ac97bc88dad0afe4dadc0770608b95d5aacf2f11/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/ovn-acl-logging/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815484    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430687858s: [/var/lib/containers/storage/overlay/8df838514665e6e27e507b914dcdc2f57a13508b4557b623dfa7fb79412fd9b0/diff /var/log/pods/openshift-ingress_e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f_e5167f1d-3718-4c96-bf19-1cdc1ab73207/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.306907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815516    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430653738s: [/var/lib/containers/storage/overlay/9012fabf6c9d37acb0fd084cebc4c8dd6d693a7cba8e42b02de88e893ac7cada/diff /var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/kube-rbac-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815538    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430674916s: [/var/lib/containers/storage/overlay/6df1c22d4bb429d567a6e21956e408dae481462cd854f2449182902dce8fee38/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg_96db7b02-b878-435a-869e-1d30e147c61d/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815560    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 54.407664953s: [/var/lib/containers/storage/overlay/e7bd185adbe2a5f6c8ee6467194d0c245d24e52db000b36e4df509e6ff6b0e8a/diff /var/log/pods/openshift-ingress_e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq_778b9a36-6a7b-4251-a91a-999e358d8c5c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815628    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430826126s: [/var/lib/containers/storage/overlay/0f64d44e017ec70c0f08927d7a5b5315efaaa27e26b52cb55a2d48db8ff58cb5/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4_18a32a7b-66fa-4f34-ab36-2beb0169e430/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815649    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430783905s: [/var/lib/containers/storage/overlay/1f335257c28f61349dc6c4495b958b214a44a29fd9b6b29ce082d223b3fa08f9/diff /var/log/pods/openshift-insights_insights-runtime-extractor-6bfcf_da9f1b66-0e1b-4c7e-a947-a897c00ccc19/extractor/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815683    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.43081607s: [/var/lib/containers/storage/overlay/815d874b17f362f32c184d09c5fa4f25b24a51f559e41679c618ca076d1f092d/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy-metrics/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815704    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430909263s: [/var/lib/containers/storage/overlay/508796c136616a6a941f39c30320d167ca9f8cc36fab3ce4085e5e6164b8c836/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr_e141d845-db9d-4f94-b05e-cfb7848e8fef/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815726    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430919654s: [/var/lib/containers/storage/overlay/801517f9d9b867bed7e9f37f0f99b4b285f8fe58d0c6fe3b4e406c4b66eb885f/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g_c0ff0cbb-4acf-4d63-9579-6aeac659b2ea/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815799    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.430959824s: [/var/lib/containers/storage/overlay/d5f8aa2b7ec7f078e5d3587bf3acd312cea2db5330296f8eeaf6747bb8ecb4af/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6_60c0e838-5c79-4bf7-a7c6-87903ae8bd8c/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307402 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815849    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.43104583s: [/var/lib/containers/storage/overlay/9f31774d55e55572596ad5b05f6824dca197d8d645a3a2091b071095f5fc348f/diff /var/log/pods/openshift-ingress_e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j_2dc48441-ccc0-47fc-abe7-7c327559eba4/istio-proxy/1.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307402 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815869    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.431026985s: [/var/lib/containers/storage/overlay/0ba4937b318aff26f5d04de663e3cbc5fb91d34ecbaf5c1a70e275fa82d80f53/diff /var/log/pods/openshift-ingress_e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z_d4161075-440d-4373-b878-57c77612c6d2/istio-proxy/1.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307402 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815890    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.431046701s: [/var/lib/containers/storage/overlay/4aaa9c0327050d64edf543a80049fd2fd46339540b39fe37be6c43a26d8f4b91/diff /var/log/pods/openshift-ingress_e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh_ef6443b3-b452-4868-9cde-f762cf522284/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307402 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815946    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.4310936s: [/var/lib/containers/storage/overlay/f62221cc1bb4ee783a4479f772f1b3eaee2ae9132e39581452d31149674b631a/diff /var/log/pods/e2e-models-e2e-worker-w3-5eedab_simulator-w3-5eedab-kserve-795c4575bb-f25x5_0c6cec58-b8d9-4856-9446-811407a747ab/main/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307566 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815969    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.431168941s: [/var/lib/containers/storage/overlay/968e208b4860faed3d3e26aa71e41adb5c447ce22ac3bb6cd22690346d2dfb5c/diff /var/log/pods/e2e-models-e2e-worker-w3-5eedab_distinct-w3-5eedab-kserve-856b8c7458-k2lgz_0874d0e8-5883-477d-b64a-d455934cfc90/main/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307566 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.815996    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.431135541s: [/var/lib/containers/storage/overlay/8e743a1c0c7cc9ef45cd9f86c51bdd235879c512aca4840e3a446be2f316bb6e/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr_f4ee5c50-5738-496a-b5ef-c86ead76e0f6/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307566 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816014    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.431167389s: [/var/lib/containers/storage/overlay/67b48a8227291881bea04d0f599da8269338873aa7b51eb6d44f387b34a084de/diff /var/log/pods/e2e-models-e2e-worker-w5-863b87_simulator-w5-863b87-kserve-bd54699cd-7g2mv_76d3e197-83a0-4793-8593-0b47b55a9fc4/main/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307566 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816032    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.431164536s: [/var/lib/containers/storage/overlay/837e2e8c2d7c95b6015e40fc5ad8bcc348a2aaccbb5d63de3e44916d84e01aef/diff /var/log/pods/e2e-models-e2e-worker-w1-120394_distinct-2-w1-120394-kserve-766594cbd8-r7nr9_57a419b6-5333-4b2c-9f01-051dc82c8707/main/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307728 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816082    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.43118173s: [/var/lib/containers/storage/overlay/79ff9c7f890e079eb19a8a8060e78ed7f0f7644efb29ded7f8d223b309ccdf2d/diff /var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k_14d43510-cfbb-4ddc-88cb-ccd4cb3a543d/main/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307728 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816106    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.431272668s: [/var/lib/containers/storage/overlay/d002b0ce8a03e77141888fb5a4a50972cf6108b7e5e59c9375aa48cad9ced5af/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx_d837dde2-3939-4400-8ea7-fee58c162983/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307728 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816125    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.431285221s: [/var/lib/containers/storage/overlay/d15dc5bffc34599d4e17b91bff8ac1423d9077ae71856a7a2755b92a3bd6dcfc/diff /var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-5knvx_8ef79a10-292b-42d9-aab0-53bee3e43ed2/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.307728 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816152    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.573777972s: [/var/lib/containers/storage/overlay/b569994ab3a7fd09901b9179b5a8ed81afe826237d0aeefc81429585cefd651a/diff /var/log/pods/e2e-models-e2e-worker-w1-120394_simulator-w1-120394-kserve-6979bf4f4f-scg5b_5734524d-67b2-4e24-a836-3ac078d5a271/main/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308021 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816171    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.431343455s: [/var/lib/containers/storage/overlay/65d81a0f3b76a7c7a9475e4c40dceab09e6a9c9e98a2e2e3c4cc474c726a6216/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45_7e09749e-e648-40a3-a5f1-f0dc35fccf24/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308021 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816188    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.431337713s: [/var/lib/containers/storage/overlay/45ab15387881b8da1f41ebe3c7b88bd293020499c9cacdb6a8c2b4a9b2ed2fb6/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg_b8847684-5676-4253-850f-9f2b2c42ea0c/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308021 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816431    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.431583576s: [/var/lib/containers/storage/overlay/07e8324700651cc1b533fd9bd9bd56d25f48721c5261994354ea99a5533a21f1/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx_38977edb-8612-48c6-a6c6-1714a50a9e4f/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308021 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816454    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.499170174s: [/var/lib/containers/storage/overlay/25ec8cf788a39d4012f5db549bcc3d440d265141a6b16d25ff0974d692c4631b/diff /var/log/pods/openshift-ingress_e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8_6aa41ecd-4a93-4b17-8e84-3f3c018c4d21/istio-proxy/1.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816483    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.499200113s: [/var/lib/containers/storage/overlay/cb4bed28be09871a074b3642832880027d77099e69a0ae8241aeb94dab27fdaf/diff /var/log/pods/openshift-image-registry_node-ca-lhwgm_00b87b36-9ce2-480a-9924-22b33256fe1e/node-ca/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816501    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.428442947s: [/var/lib/containers/storage/overlay/6ea0c1ff6b4e328bbe49837fbf2858fecebf87243389fc2d2d8ab2b78934c93e/diff /var/log/pods/openshift-monitoring_telemeter-client-79c994cf5c-ckj5h_e065a5c3-95fd-4ac8-a287-8e1d20219201/telemeter-client/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816519    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.428459893s: [/var/lib/containers/storage/overlay/92e435ce4ca5afe43253221b15bf185141e8b9ccff9215e533a29abb68c8d01e/diff /var/log/pods/openshift-network-operator_iptables-alerter-dlp6w_6767bd29-3165-4e67-81b3-e320e8324034/iptables-alerter/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816570    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.428507041s: [/var/lib/containers/storage/overlay/49fe87eab95fb5bcdb653a69186242a7bf84794358441a45b2ffc81148fc7f68/diff /var/log/pods/openshift-multus_network-metrics-daemon-tk8ht_dd733778-7002-4a26-a4e8-4525d8f6bee9/network-metrics-daemon/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308344 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816589    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.428526812s: [/var/lib/containers/storage/overlay/b8b56f59390f731005dd4f5e142899c74da144d464f61fbeea461b527f3dfc3d/diff /var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/prom-label-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308344 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816607    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.428542898s: [/var/lib/containers/storage/overlay/2baaae2b22b3f8e778df214acf8cc253c7a7bfa34a8e2a49f4c49e86ff018e9a/diff /var/log/pods/openshift-console_downloads-6dd789c8b7-pgmzg_6e78d6c7-c325-48f2-af90-804dade84d8f/download-server/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308344 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816633    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.4285688s: [/var/lib/containers/storage/overlay/d5d50d36d455b3124bba1a4682c6b556a344e4c0f72561f6ab14472bbe327aa9/diff /var/log/pods/openshift-multus_multus-additional-cni-plugins-ngkbz_c2d7ba52-9a2b-4871-aca0-3c4e1934c572/kube-multus-additional-cni-plugins/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308344 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816651    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.428584972s: [/var/lib/containers/storage/overlay/613c01497ad5e2672a318b68ad56355e93f5e704f0918037c43a2d38d967a924/diff /var/log/pods/openshift-monitoring_telemeter-client-79c994cf5c-ckj5h_e065a5c3-95fd-4ac8-a287-8e1d20219201/reload/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816669    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.428602103s: [/var/lib/containers/storage/overlay/48a8a99b5aa38b5a1f8af93f2cf91f2afe257cea7b9b3fc9ad86c404aea4cfa8/diff /var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816688    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.428619458s: [/var/lib/containers/storage/overlay/df02c5a738481c502ee0fbb85ca8fb2c47529fde2b1bfbb73efe51cb364c310b/diff /var/log/pods/openshift-ovn-kubernetes_ovnkube-node-44l94_c86c4f40-2978-40f0-bfc1-c3cfffca0d8c/ovn-controller/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816740    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.428499365s: [/var/lib/containers/storage/overlay/7b63acde62291120f32b79155fd4ac7703b60aa331f58870ab39e3003a8008d6/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb_cf85b137-2b1e-4e22-8f6e-744a02ea4e46/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816758    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.428517117s: [/var/lib/containers/storage/overlay/4832bee5d30fb081704a3c96e43d22d7214d83e63b2bbc7c3a50de9c878d8550/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46_806cc8ce-f608-4975-94dd-5edade11fc8c/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308665 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816784    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.431909968s: [/var/lib/containers/storage/overlay/f04694bfe7aed9c3a051f679dbed90340f776045c06d3bc39068d91ebc64b1d9/diff /var/log/pods/openshift-dns_dns-default-zmrdt_f59eba5d-a500-462e-9ecd-4ef9e553f873/dns/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308665 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816806    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.104517013s: [/var/lib/containers/storage/overlay/001563ee2fc791e222cb10a7b111cdb2621004a364564c35e4421a7c57d2db0b/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m_9051e10f-854f-4cc8-90a7-dda319c49c9a/maas-api/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.308665 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816865    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.428622082s: [/var/lib/containers/storage/overlay/3b4eb99b514a85975e9e184eb65b4fc878deddd7205c795539ae51ac4de5caab/diff /var/log/pods/openshift-ingress_payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj_bca2dabc-94e2-4875-ba3f-fa9332afe8dc/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308665 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816884    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.278552065s: [/var/lib/containers/storage/overlay/38e2db64053a75d5e267a571b5c8029e880b65c662fc935cba8b4b78f3031bad/diff /var/log/pods/openshift-ingress_e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc_ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0/istio-proxy/1.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816902    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.074491835s: [/var/lib/containers/storage/overlay/fa8831aafda2d92f0ce3f59ac079b571f74f757924d7c3247d164db1d6711b3c/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j_e5ae7b61-cd5f-4a38-b038-28aaa6991eb7/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816931    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.074523461s: [/var/lib/containers/storage/overlay/2773d507773300fe9114d7cad1ed33c15472da65f5700864c7b4d36c59438123/diff /var/log/pods/odh-ai-gateway-infra_maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75_7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f/maas-api/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.308840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816949    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 35.957274683s: [/var/lib/containers/storage/overlay/be0a87d2eccccae8e85688b90a61b88439e6fd706c76be4ba52b6f95c4beac73/diff /var/log/pods/openshift-ingress_e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz_408afe2d-75a0-4537-b58c-9ab31f161685/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.308840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.816968    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m8.791501732s: [/var/lib/containers/storage/overlay/985ba12eb48566836f0fcad469e6eeb2711b77cf3eca5b054713e9dd9c612af3/diff /var/log/pods/openshift-ingress_kube-auth-proxy-78648788bf-npsk7_347c8613-0791-494e-83e3-faded5593d88/kube-auth-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.309007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.817032    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m8.792132764s: [/var/lib/containers/storage/overlay/d0ef0428e1dc75ad127901851f1b2fc725d21cd1b094959f49dcd7d557ab4f1a/diff /var/log/pods/openshift-ingress_e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths_5775b5a6-6407-46f4-84ef-96c708d7934b/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.309007 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:00.817171    4456 logging.go:55] [core] [Channel #6615 SubChannel #6642]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.232:50052", ServerName: "10.134.0.232:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:38228->10.134.0.232:50052: use of closed network connection"
Nov 02 18:11:11.309007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.817274    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.499997126s: [/var/lib/containers/storage/overlay/1a394e1eaa16f81c47d4d9f50cab7c43ca075ce2f22fa2906b7916363672c95c/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.309007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.818018    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.500732129s: [/var/lib/containers/storage/overlay/715bfa0d02500a7607ee5ba949c9655d2d098909dc187d3fa64c431aa173e55c/diff /var/log/pods/openshift-ingress_payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb_cc682b81-5f9e-4cd1-98ae-d32e89878ea3/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.309007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.818071    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.430043441s: [/var/lib/containers/storage/overlay/1a8ec8fdfc84ff22ef5720fbfd7b513681a22a4a90a63c80b66dec40cf96c5a9/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.309209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.818132    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.430103677s: [/var/lib/containers/storage/overlay/b08034c65637061c7d06856c8763d8af38fa0083db3da8a46ee71187bc8aa3f5/diff ]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.309209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.825475    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m8.800630278s: [/var/lib/containers/storage/overlay/ffc640e61d9f2b8dc312d9f010b835c9e91e982e4f9cff0f120f887c46ef8f3a/diff /var/log/pods/openshift-ingress_e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb_551602c4-254b-4532-8961-a98bf0f81089/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.309209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.827226    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.289927828s: [/var/lib/containers/storage/overlay/54eb6dcdb0f244dbd1ab517bf5fd9b7b8ca4a049b4741da30b72d995007c9bd3/diff /var/log/pods/openshift-ingress_payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g_44b66f41-99ae-446b-a50a-2ec0b71563a4/payload-pre-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.309209 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:00.833924    4456 logging.go:55] [core] [Channel #6620 SubChannel #6673]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.43:50052", ServerName: "10.134.0.43:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:52310->10.134.0.43:50052: use of closed network connection"
Nov 02 18:11:11.309209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.834380    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.122065256s: [/var/lib/containers/storage/overlay/7957e0d146f8847395c58adcac4b12e2b0fbb5573d7580c6bc5eea896ab35073/diff /var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-6jpq8_a1ea3dac-7073-4564-b247-89ea94a64342/manager/1.log]; will not log again for this container unless duration exceeds 3s
Nov 02 18:11:11.309411 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:00.835976    4456 logging.go:55] [core] [Channel #6658 SubChannel #6663]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.170:50052", ServerName: "10.134.0.170:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:34112->10.134.0.170:50052: use of closed network connection"
Nov 02 18:11:11.309411 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.859116    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.474238361s: [/var/lib/containers/storage/overlay/735f0e10d9df130fc8a5a1c199498247b93c3829feaab94f9cf3d5136c7492bd/diff /var/log/pods/openshift-image-registry_image-registry-59767d8795-7vwkt_0db36d92-4712-4338-958a-ebabf2c137ba/registry/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.309411 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.893875    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.309411 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.893923    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.224:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.309411 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.898591    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vrvf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.309613 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.898630    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.309613 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.899483    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.309613 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.899517    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.219:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.309613 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.900026    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.309613 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.900058    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.197:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.309830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.904707    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.366369502s: [/var/lib/containers/storage/overlay/ad9f3bb11ff9cab22055aad521309605cbe31ca829c1820e58022c92cded277f/diff /var/log/pods/openshift-ingress_payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6_1c190514-ba33-4844-a9ec-e4bb47b5f706/payload-processing/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.309830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.906283    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.163859377s: [/var/lib/containers/storage/overlay/04144c7db0ef8494e8777526c23ed619a940be7d263e9bd174a3253037d6d77a/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d_8cba3353-fa51-41d3-acb4-a3cfbaae1daa/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.309830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.952440    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.309830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.952493    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.309995 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:00.990441    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.111906717s: [/var/lib/containers/storage/overlay/8c0ca4c9e4e4e93687de6c7e703ba32222bc3330b9cf5dbd83a5cbdfbc1eb1a4/diff /var/log/pods/openshift-ingress_e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r_67b961eb-ab19-4d8d-bb2a-db7c94a93436/istio-proxy/1.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.309995 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.092479    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" podUID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.232:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.309995 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:01.133012    4456 logging.go:55] [core] [Channel #6668 SubChannel #6701]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.232:50052", ServerName: "10.134.0.232:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:38248->10.134.0.232:50052: use of closed network connection"
Nov 02 18:11:11.309995 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.136703    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m9.111804935s: [/var/lib/containers/storage/overlay/4c8d263b1f9b1e5f7f61488e19a80e3bdfd0a74fc133c4635b7c83f9383972d5/diff /var/log/pods/openshift-ingress_e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm_dbd34da0-9b90-458d-9581-ed9caf3daa11/istio-proxy/1.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.310161 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.167120    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 36.424705305s: [/var/lib/containers/storage/overlay/8e3943818fa8cc19ef6f82354867dfbe856e505e34cf4a7d05bc429843ee18e5/diff /var/log/pods/kuadrant-system_authorino-6f5b7cd5c7-brdvs_040ec9bf-319e-4bae-b233-4e4e7e74a379/authorino/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.310161 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.171089    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 37.475868203s: [/var/lib/containers/storage/overlay/c054c6b27f869c3385aeca4dc9bbc0f3c6a68aec5c08b24d9bfc20cfd651da17/diff /var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-xc9kg_3b0e614e-16b1-4e28-bea6-4b7b9a62b177/main/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.310161 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.219374    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m7.819368934s: [/var/lib/containers/storage/overlay/48c65928388ea29afd8ce97d9eec05ed39a6fd3da709a73d2174493d7947d9f4/diff /var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-c8vr7_08c9aa32-dadc-4112-b608-4cc64f56295c/kuadrant-console-plugin/0.log]; will not log again for this container unless duration exceeds 5s
Nov 02 18:11:11.310161 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.232049    4456 patch_prober.go:29] interesting pod/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.310161 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.232103    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.181:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.310392 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.248303    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 1m9.223511726s: [/var/lib/containers/storage/overlay/d09ba0e972daf0879be4ba9795c180ae5ce98f9c793ff54605297f1c7dd5eb3d/diff /var/log/pods/openshift-ingress_e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88_cb09170f-1b12-4a26-8e5b-c5e2c2964a4d/istio-proxy/1.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.310392 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.249555    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 51.864678036s: [/var/lib/containers/storage/overlay/a143e700ec1dc2f1d0157b0b73e4385b467977554b146f0849b335957d53870f/diff /var/log/pods/openshift-ingress_e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7_73849b3d-673e-41a8-aff1-db9f0a9cce85/istio-proxy/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.310392 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.476612    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg" podUID="3b0e614e-16b1-4e28-bea6-4b7b9a62b177" containerName="main" probeResult="failure" output="Get \"https://10.134.0.30:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.310392 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.476598    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.310392 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.477697    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" probeResult="failure" output="Get \"https://10.134.0.66:8000/health\": context deadline exceeded"
Nov 02 18:11:11.310594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.477712    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4" podUID="6e44698a-838e-4eea-b956-585673963ed9" containerName="main" probeResult="failure" output="Get \"https://10.134.0.32:8000/health\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.310594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.579845    4456 trace.go:236] Trace[1606831645]: "Calculate volume metrics of tmp for pod openshift-console/downloads-6dd789c8b7-pgmzg" (02-Nov-2026 18:10:24.552) (total time: 37027ms):
Nov 02 18:11:11.310594 ip-10-0-141-87 kubenswrapper[4456]: Trace[1606831645]: [37.027274562s] [37.027274562s] END
Nov 02 18:11:11.310594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.586733    4456 fsHandler.go:133] fs: disk usage and inodes count on following dirs took 52.201876538s: [/var/lib/containers/storage/overlay/6711a046696ca85a051e460f8eca0f62ee6736867b06bec3453afa502df3ff47/diff /var/log/pods/cert-manager_cert-manager-cainjector-5c57bbd4fc-9mjv9_9fe1660e-3733-4c7f-bc5c-3820d84aa143/cert-manager-cainjector/0.log]; will not log again for this container unless duration exceeds 4s
Nov 02 18:11:11.310594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.639135    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.187:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.310594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.640143    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.173:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.310867 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.665978    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.310867 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:01.679981    4456 logging.go:55] [core] [Channel #6672 SubChannel #6675]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.187:50052", ServerName: "10.134.0.187:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:50452->10.134.0.187:50052: use of closed network connection"
Nov 02 18:11:11.310867 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:01.680999    4456 logging.go:55] [core] [Channel #6674 SubChannel #6677]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.173:50052", ServerName: "10.134.0.173:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:44290->10.134.0.173:50052: use of closed network connection"
Nov 02 18:11:11.310867 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.703758    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5knvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.310867 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.703776    4456 patch_prober.go:29] interesting pod/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.311073 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.703830    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx" podUID="8ef79a10-292b-42d9-aab0-53bee3e43ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.311073 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.703868    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.56:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.311073 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:01.706986    4456 logging.go:55] [core] [Channel #6682 SubChannel #6683]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.171:50052", ServerName: "10.134.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:45500->10.134.0.171:50052: use of closed network connection"
Nov 02 18:11:11.311073 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.721943    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.43:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.311073 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.723044    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.42:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.311274 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.724634    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" podUID="22b556c1-112b-49d4-932c-0143e912a54e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.186:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.311274 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.725227    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.129:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.311274 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.725623    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" podUID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.172:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.311274 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.737942    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.311274 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.742968    4456 patch_prober.go:29] interesting pod/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.311477 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.743004    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.194:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.311477 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.743964    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.311477 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.743998    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.39:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.311477 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:01.763635    4456 logging.go:55] [core] [Channel #6686 SubChannel #6687]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.43:50052", ServerName: "10.134.0.43:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:52322->10.134.0.43:50052: use of closed network connection"
Nov 02 18:11:11.311477 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:01.764002    4456 logging.go:55] [core] [Channel #6689 SubChannel #6690]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.42:50052", ServerName: "10.134.0.42:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:44400->10.134.0.42:50052: use of closed network connection"
Nov 02 18:11:11.311682 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:01.764960    4456 logging.go:55] [core] [Channel #6691 SubChannel #6692]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.186:50052", ServerName: "10.134.0.186:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:54848->10.134.0.186:50052: use of closed network connection"
Nov 02 18:11:11.311682 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:01.765969    4456 logging.go:55] [core] [Channel #6693 SubChannel #6695]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.129:50052", ServerName: "10.134.0.129:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:58062->10.134.0.129:50052: use of closed network connection"
Nov 02 18:11:11.311682 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:01.765998    4456 logging.go:55] [core] [Channel #6694 SubChannel #6696]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.172:50052", ServerName: "10.134.0.172:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:41820->10.134.0.172:50052: use of closed network connection"
Nov 02 18:11:11.311682 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.900982    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-vrvf2 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.311862 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:01.901034    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.113:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.311862 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:02.035410    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" podUID="d837dde2-3939-4400-8ea7-fee58c162983" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.116:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.311862 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:02.035517    4456 logging.go:55] [core] [Channel #6703 SubChannel #6704]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.116:50052", ServerName: "10.134.0.116:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:33232->10.134.0.116:50052: use of closed network connection"
Nov 02 18:11:11.311862 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:02.497230    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.176:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.311862 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:02.497967    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.175:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.312070 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:02.539916    4456 logging.go:55] [core] [Channel #6707 SubChannel #6708]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.175:50052", ServerName: "10.134.0.175:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:33380->10.134.0.175:50052: use of closed network connection"
Nov 02 18:11:11.312070 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:02.540031    4456 logging.go:55] [core] [Channel #6706 SubChannel #6709]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.176:50052", ServerName: "10.134.0.176:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:50092->10.134.0.176:50052: use of closed network connection"
Nov 02 18:11:11.312070 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:02.562195    4456 trace.go:236] Trace[1466703999]: "Calculate volume metrics of ca-trust-extracted for pod openshift-image-registry/image-registry-59767d8795-7vwkt" (02-Nov-2026 18:11:00.724) (total time: 1837ms):
Nov 02 18:11:11.312070 ip-10-0-141-87 kubenswrapper[4456]: Trace[1466703999]: [1.837842144s] [1.837842144s] END
Nov 02 18:11:11.312070 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:02.723244    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.42:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.312070 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:02.725338    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" podUID="22b556c1-112b-49d4-932c-0143e912a54e" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.186:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.312310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:02.725476    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.129:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.312310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:02.726351    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" podUID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.172:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.312310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:02.743112    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" podUID="e141d845-db9d-4f94-b05e-cfb7848e8fef" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.180:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.312310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:02.762690    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.170:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.312310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:02.762724    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing" probeResult="failure" output="timeout: failed to connect service \"10.134.0.171:50052\" within 1s: context deadline exceeded"
Nov 02 18:11:11.312510 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:02.764033    4456 logging.go:55] [core] [Channel #6712 SubChannel #6713]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.42:50052", ServerName: "10.134.0.42:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:36098->10.134.0.42:50052: use of closed network connection"
Nov 02 18:11:11.312510 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:02.765967    4456 logging.go:55] [core] [Channel #6715 SubChannel #6716]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.186:50052", ServerName: "10.134.0.186:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:37724->10.134.0.186:50052: use of closed network connection"
Nov 02 18:11:11.312510 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:02.765976    4456 logging.go:55] [core] [Channel #6718 SubChannel #6719]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.129:50052", ServerName: "10.134.0.129:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:48138->10.134.0.129:50052: use of closed network connection"
Nov 02 18:11:11.312510 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:02.767029    4456 logging.go:55] [core] [Channel #6720 SubChannel #6722]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.172:50052", ServerName: "10.134.0.172:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:41462->10.134.0.172:50052: use of closed network connection"
Nov 02 18:11:11.312673 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:02.785040    4456 logging.go:55] [core] [Channel #6724 SubChannel #6725]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.180:50052", ServerName: "10.134.0.180:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:58818->10.134.0.180:50052: use of closed network connection"
Nov 02 18:11:11.312673 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:02.802947    4456 logging.go:55] [core] [Channel #6728 SubChannel #6730]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.170:50052", ServerName: "10.134.0.170:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:46962->10.134.0.170:50052: use of closed network connection"
Nov 02 18:11:11.312673 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:02.804941    4456 logging.go:55] [core] [Channel #6727 SubChannel #6729]grpc: addrConn.createTransport failed to connect to {Addr: "10.134.0.171:50052", ServerName: "10.134.0.171:50052", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read tcp 10.134.0.2:45050->10.134.0.171:50052: use of closed network connection"
Nov 02 18:11:11.312673 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:03.100956    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" podUID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" containerName="payload-processing" probeResult="failure" output="timeout: health rpc did not complete within 1s"
Nov 02 18:11:11.312673 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:04.242932    4456 trace.go:236] Trace[215087479]: "iptables ChainExists" (02-Nov-2026 18:10:59.516) (total time: 4726ms):
Nov 02 18:11:11.312673 ip-10-0-141-87 kubenswrapper[4456]: Trace[215087479]: [4.726215082s] [4.726215082s] END
Nov 02 18:11:11.312923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:04.515944    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="kube-system/konnectivity-agent-mxsjq" podUID="367d8d85-a5e8-462d-b1d7-91a01e62b726" containerName="konnectivity-agent" probeResult="failure" output="Get \"http://127.0.0.1:2041/healthz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.312923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:05.848031    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:11:11.312923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:05.848087    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": net/http: request canceled (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.312923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:06.863964    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.312923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:06.877110    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": dial tcp 10.134.0.28:8081: connect: connection refused" start-of-body=
Nov 02 18:11:11.313121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:06.877154    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": dial tcp 10.134.0.28:8081: connect: connection refused"
Nov 02 18:11:11.313121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.041803    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:11.313121 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.163089    4456 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="5m41.556s"
Nov 02 18:11:11.313121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.163173    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 18:11:11.313121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.163227    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 18:11:11.313121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.163245    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"]
Nov 02 18:11:11.313121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.163635    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:11:11.313121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.342242    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-c504605b-6fqhs"]
Nov 02 18:11:11.313121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.349076    4456 patch_prober.go:29] interesting pod/jobset-controller-manager-77cfb5f94-gfbr9 container/manager namespace/openshift-jobset-operator: Readiness probe status=failure output="Get \"http://10.134.0.28:8081/readyz\": dial tcp 10.134.0.28:8081: connect: connection refused" start-of-body=
Nov 02 18:11:11.313449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.349122    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.28:8081/readyz\": dial tcp 10.134.0.28:8081: connect: connection refused"
Nov 02 18:11:11.313449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.355573    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-872354-dockercfg-hw7xl\""
Nov 02 18:11:11.313449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.357547    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a5c1da-openshift-default-dockercfg-gjtlc\""
Nov 02 18:11:11.313449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.358116    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"default-dockercfg-f7th6\""
Nov 02 18:11:11.313449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.360921    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" secret="" err="secret \"e2e-inf-a-a5c1da-openshift-default-dockercfg-gjtlc\" not found"
Nov 02 18:11:11.313449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.360983    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" secret="" err="secret \"payload-processing-e2e-inf-a-872354-dockercfg-hw7xl\" not found"
Nov 02 18:11:11.313449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.367511    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gml5d\""
Nov 02 18:11:11.313449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.367552    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-node-identity-dockercfg-qbqb7\""
Nov 02 18:11:11.313449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.370475    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w0-069fa5-openshift-default-dockercfg-ckxtj\""
Nov 02 18:11:11.313828 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.371256    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a29999-dockercfg-r7mt7\""
Nov 02 18:11:11.313828 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.371581    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-dockercfg-nv7lh\""
Nov 02 18:11:11.313828 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.371620    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-dockercfg-rdt6t\""
Nov 02 18:11:11.313828 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.371665    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"e4828365cff34649b08e97b25ad67554fb1835933f1008be8e43bed2f20e7560"} pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" containerMessage="Container maas-api failed liveness probe, will be restarted"
Nov 02 18:11:11.313828 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.371675    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-d5b905-openshift-default-dockercfg-vtgjv\""
Nov 02 18:11:11.313828 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.371708    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" containerID="cri-o://e4828365cff34649b08e97b25ad67554fb1835933f1008be8e43bed2f20e7560" gracePeriod=30
Nov 02 18:11:11.313828 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.372578    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-8a209f-openshift-default-dockercfg-whvdh\""
Nov 02 18:11:11.313828 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.379234    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" secret="" err="secret \"payload-processing-e2e-inf-a-a29999-dockercfg-r7mt7\" not found"
Nov 02 18:11:11.314153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.380493    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" secret="" err="secret \"payload-processing-e2e-inf-a-a29999-dockercfg-r7mt7\" not found"
Nov 02 18:11:11.314153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.380525    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"a3e5b342fbd24d60a087ce776f7958683a916f3ab12b9d0f407b62ce53f0987a"} pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" containerMessage="Container manager failed liveness probe, will be restarted"
Nov 02 18:11:11.314153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.380581    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" containerID="cri-o://a3e5b342fbd24d60a087ce776f7958683a916f3ab12b9d0f407b62ce53f0987a" gracePeriod=10
Nov 02 18:11:11.314153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.382485    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"default-dockercfg-p5nzm\""
Nov 02 18:11:11.314153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.382490    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-96768b-dockercfg-dm54d\""
Nov 02 18:11:11.314153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.384110    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-cainjector-dockercfg-x9lxq\""
Nov 02 18:11:11.314153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.384129    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-5fd3e0-openshift-default-dockercfg-kxlxh\""
Nov 02 18:11:11.314153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.388730    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" secret="" err="secret \"payload-processing-e2e-inf-a-96768b-dockercfg-dm54d\" not found"
Nov 02 18:11:11.314487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.390031    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" secret="" err="secret \"payload-processing-e2e-inf-a-96768b-dockercfg-dm54d\" not found"
Nov 02 18:11:11.314487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.419738    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-96768b-openshift-default-dockercfg-v8qql\""
Nov 02 18:11:11.314487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.419964    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w3-227939-dockercfg-4rtbn\""
Nov 02 18:11:11.314487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.419966    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-fbbbf8-openshift-default-dockercfg-b9pj2\""
Nov 02 18:11:11.314487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.422471    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w4-2899ab-openshift-default-dockercfg-t6vss\""
Nov 02 18:11:11.314487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.422501    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-2c9b1d-dockercfg-tk2wk\""
Nov 02 18:11:11.314487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.422534    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-authorino-dockercfg-kqm5s\""
Nov 02 18:11:11.314487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.422535    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-dockercfg-qn5gh\""
Nov 02 18:11:11.314487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.423842    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"global-pull-secret-syncer-dockercfg-zt2z5\""
Nov 02 18:11:11.314487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.423950    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w1-120394-openshift-default-dockercfg-qq496\""
Nov 02 18:11:11.314487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.424179    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"default-dockercfg-h47km\""
Nov 02 18:11:11.317217 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.424286    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-b-w0-c4dbe3-dockercfg-qp9g8\""
Nov 02 18:11:11.317217 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.425275    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" secret="" err="secret \"e2e-inf-a-96768b-openshift-default-dockercfg-v8qql\" not found"
Nov 02 18:11:11.317217 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.426870    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"node-ca-dockercfg-bl7zd\""
Nov 02 18:11:11.317217 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.428147    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" secret="" err="secret \"payload-processing-e2e-inf-b-2c9b1d-dockercfg-tk2wk\" not found"
Nov 02 18:11:11.317217 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.428841    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" secret="" err="secret \"e2e-inf-a-96768b-openshift-default-dockercfg-v8qql\" not found"
Nov 02 18:11:11.317217 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.430168    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"default-dockercfg-8x5zb\""
Nov 02 18:11:11.317217 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.431252    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317"} pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" containerMessage="Container maas-api failed liveness probe, will be restarted"
Nov 02 18:11:11.317551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.431296    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" containerID="cri-o://e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317" gracePeriod=30
Nov 02 18:11:11.317551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.432535    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-kubernetes-node-dockercfg-jl8vw\""
Nov 02 18:11:11.317551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.435659    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-console\"/\"downloads-dockercfg-gclvz\""
Nov 02 18:11:11.317551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.435762    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"default-dockercfg-q6667\""
Nov 02 18:11:11.317551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.437912    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w3-557feb-openshift-default-dockercfg-lvqfg\""
Nov 02 18:11:11.317551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.439037    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w3-5eedab-openshift-default-dockercfg-57c6m\""
Nov 02 18:11:11.317551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.441095    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="download-server" containerStatusID={"Type":"cri-o","ID":"4cee89e694b5799548d6085cbcaa2612948ac905b236f27ac06a462827cad81c"} pod="openshift-console/downloads-6dd789c8b7-pgmzg" containerMessage="Container download-server failed liveness probe, will be restarted"
Nov 02 18:11:11.317551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.441140    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server" containerID="cri-o://4cee89e694b5799548d6085cbcaa2612948ac905b236f27ac06a462827cad81c" gracePeriod=5
Nov 02 18:11:11.317880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.441496    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-a-a5c1da-dockercfg-vlnwc\""
Nov 02 18:11:11.317880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.441575    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"maas-default-gateway-openshift-default-dockercfg-lkf8q\""
Nov 02 18:11:11.317880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.445959    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"jobset-controller-manager-dockercfg-bfthl\""
Nov 02 18:11:11.317880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.446004    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w3-557feb-dockercfg-nfpss\""
Nov 02 18:11:11.317880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.446049    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-diagnostics\"/\"network-check-target-dockercfg-ppwdb\""
Nov 02 18:11:11.317880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.446853    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" secret="" err="secret \"payload-processing-e2e-inf-a-a5c1da-dockercfg-vlnwc\" not found"
Nov 02 18:11:11.317880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.448466    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" secret="" err="secret \"payload-processing-e2e-inf-a-872354-dockercfg-hw7xl\" not found"
Nov 02 18:11:11.317880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.448866    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"50594c76d72639934489d5c1e00e5b464a34d6a50aa2d64165ad25bf4c978a07"} pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" containerMessage="Container manager failed liveness probe, will be restarted"
Nov 02 18:11:11.318210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.448925    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" podUID="6957a218-62fc-4dd8-892e-4477b08c046f" containerName="manager" containerID="cri-o://50594c76d72639934489d5c1e00e5b464a34d6a50aa2d64165ad25bf4c978a07" gracePeriod=10
Nov 02 18:11:11.318210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.449211    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w4-f59572-dockercfg-s5jw7\""
Nov 02 18:11:11.318210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.456027    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.318210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.456198    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-872354\""
Nov 02 18:11:11.318210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.458166    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w6-b91583-dockercfg-vbwks\""
Nov 02 18:11:11.318210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.461247    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w2-f0a3b8-dockercfg-h2cjr\""
Nov 02 18:11:11.318210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.461512    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"node-resolver-dockercfg-9lngf\""
Nov 02 18:11:11.318210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.461610    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"ingress-canary-dockercfg-rj9zp\""
Nov 02 18:11:11.318210 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.461638    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-872354: configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.318566 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.461705    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume podName:cf85b137-2b1e-4e22-8f6e-744a02ea4e46 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:07.96168359 +0000 UTC m=+2484.452746746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume") pod "payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" (UID: "cf85b137-2b1e-4e22-8f6e-744a02ea4e46") : configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.318566 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.461776    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-2c9b1d-openshift-default-dockercfg-v7bnw\""
Nov 02 18:11:11.318566 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.461907    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-dockercfg-j2fn5\""
Nov 02 18:11:11.318566 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.462743    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" secret="" err="secret \"payload-processing-e2e-inf-b-2c9b1d-dockercfg-tk2wk\" not found"
Nov 02 18:11:11.318566 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.463306    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-sa-dockercfg-fw7vt\""
Nov 02 18:11:11.318566 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.463675    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-dockercfg-5nstj\""
Nov 02 18:11:11.318566 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.463695    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-dockercfg-mhkk4\""
Nov 02 18:11:11.318566 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.465525    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-fbbbf8-dockercfg-29vsx\""
Nov 02 18:11:11.342954 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.466280    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"registry-dockercfg-mnv5k\""
Nov 02 18:11:11.342954 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.466433    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="kube-rbac-proxy-web" containerStatusID={"Type":"cri-o","ID":"61f0d172af153366e5fe0177879402cf8b046838c4d8800dd144c8d0095dd565"} pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" containerMessage="Container kube-rbac-proxy-web failed liveness probe, will be restarted"
Nov 02 18:11:11.342954 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.466619    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" containerID="cri-o://61f0d172af153366e5fe0177879402cf8b046838c4d8800dd144c8d0095dd565" gracePeriod=120
Nov 02 18:11:11.342954 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.469472    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="prometheus-operator-admission-webhook" containerStatusID={"Type":"cri-o","ID":"5b3b2a55035fb65589abdc8d0eb062b82d79bbb8b490a8438c8c6505f3ee9c32"} pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" containerMessage="Container prometheus-operator-admission-webhook failed liveness probe, will be restarted"
Nov 02 18:11:11.342954 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.469514    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" podUID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerName="prometheus-operator-admission-webhook" containerID="cri-o://5b3b2a55035fb65589abdc8d0eb062b82d79bbb8b490a8438c8c6505f3ee9c32" gracePeriod=30
Nov 02 18:11:11.342954 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.471769    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" secret="" err="secret \"payload-processing-e2e-inf-b-fbbbf8-dockercfg-29vsx\" not found"
Nov 02 18:11:11.343273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.471894    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="registry" containerStatusID={"Type":"cri-o","ID":"2501b584d5ee7323484f50ac5f32dd50c7fef23cf98bd9bf3120550c1d0985af"} pod="openshift-image-registry/image-registry-59767d8795-7vwkt" containerMessage="Container registry failed liveness probe, will be restarted"
Nov 02 18:11:11.343273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.475931    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-a29999-openshift-default-dockercfg-6b7bh\""
Nov 02 18:11:11.343273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.476108    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"default-dockercfg-8dv9q\""
Nov 02 18:11:11.343273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.476361    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.343273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.477668    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-dockercfg-mw4s9\""
Nov 02 18:11:11.343273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.477923    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"multus-ancillary-tools-dockercfg-xdl26\""
Nov 02 18:11:11.343273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.478032    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-8a209f-dockercfg-g8ztf\""
Nov 02 18:11:11.343273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.481613    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" secret="" err="secret \"e2e-inf-a-a29999-openshift-default-dockercfg-6b7bh\" not found"
Nov 02 18:11:11.343273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.490695    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" secret="" err="secret \"payload-processing-e2e-inf-b-fbbbf8-dockercfg-29vsx\" not found"
Nov 02 18:11:11.354715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.493872    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-a-872354-openshift-default-dockercfg-rdxd7\""
Nov 02 18:11:11.354715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.494299    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-w5-863b87-openshift-default-dockercfg-5mwm4\""
Nov 02 18:11:11.354715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.494578    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dklz6\""
Nov 02 18:11:11.354715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.495672    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p2jm6\""
Nov 02 18:11:11.354715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.498971    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" secret="" err="secret \"e2e-inf-a-872354-openshift-default-dockercfg-rdxd7\" not found"
Nov 02 18:11:11.354715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.499125    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" secret="" err="secret \"e2e-inf-a-96768b-openshift-default-dockercfg-v8qql\" not found"
Nov 02 18:11:11.354715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.499866    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" secret="" err="secret \"e2e-inf-a-872354-openshift-default-dockercfg-rdxd7\" not found"
Nov 02 18:11:11.354715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.500933    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-dockercfg-sn4p8\""
Nov 02 18:11:11.354979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.501126    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="manager" containerStatusID={"Type":"cri-o","ID":"f46a45f18a610493bf46d8787ee563938a877ab2e179a2e516f44f6e70fcc573"} pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" containerMessage="Container manager failed liveness probe, will be restarted"
Nov 02 18:11:11.354979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.501173    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" containerID="cri-o://f46a45f18a610493bf46d8787ee563938a877ab2e179a2e516f44f6e70fcc573" gracePeriod=10
Nov 02 18:11:11.354979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.513507    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-w4-2899ab-dockercfg-lwvws\""
Nov 02 18:11:11.354979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.522429    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-sa-dockercfg-nn2p2\""
Nov 02 18:11:11.354979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.522526    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"tuned-dockercfg-vjnnw\""
Nov 02 18:11:11.354979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.524797    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-dockercfg-4b9fl\""
Nov 02 18:11:11.354979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.529156    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" secret="" err="secret \"payload-processing-e2e-inf-a-a5c1da-dockercfg-vlnwc\" not found"
Nov 02 18:11:11.354979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.546311    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-dockercfg-4679w\""
Nov 02 18:11:11.363607 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.555763    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="maas-api" containerStatusID={"Type":"cri-o","ID":"b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436"} pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" containerMessage="Container maas-api failed liveness probe, will be restarted"
Nov 02 18:11:11.363607 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.555836    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" containerID="cri-o://b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436" gracePeriod=30
Nov 02 18:11:11.363607 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.556911    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-inf-b-b2fc9f-openshift-default-dockercfg-tqz7r\""
Nov 02 18:11:11.363607 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.558733    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"cni-copy-resources\""
Nov 02 18:11:11.363607 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.558971    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.363607 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.558968    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a5c1da\""
Nov 02 18:11:11.363607 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.559020    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w3-557feb\""
Nov 02 18:11:11.363607 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.559685    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"multus-daemon-config\""
Nov 02 18:11:11.363607 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.560095    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w4-f59572\""
Nov 02 18:11:11.363607 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.560644    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.364038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.560863    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w0-c4dbe3\""
Nov 02 18:11:11.364038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.561791    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-controller-manager-service-cert\""
Nov 02 18:11:11.364038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.562070    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.364038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.562297    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-2c9b1d\""
Nov 02 18:11:11.364038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.562310    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"opendatahub-operator-system\"/\"opendatahub-operator-metrics-tls\""
Nov 02 18:11:11.364038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.562605    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-rules\""
Nov 02 18:11:11.364038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.562640    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"openshift-gw-ca-root-cert\""
Nov 02 18:11:11.364038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.562684    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"facebook-opt-125m-simulated-kserve-self-signed-certs\""
Nov 02 18:11:11.364038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.562726    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-tls\""
Nov 02 18:11:11.364038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.563152    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.364038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.563287    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w6-b91583\""
Nov 02 18:11:11.364038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.563322    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"image-registry-certificates\""
Nov 02 18:11:11.364558 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.563374    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a5c1da: configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.364558 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.563420    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume podName:96db7b02-b878-435a-869e-1d30e147c61d nodeName:}" failed. No retries permitted until 2026-11-02 18:11:08.063405075 +0000 UTC m=+2484.554468244 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" (UID: "96db7b02-b878-435a-869e-1d30e147c61d") : configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.364558 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565025    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"jobset-manager-config\""
Nov 02 18:11:11.364558 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565059    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-96768b\""
Nov 02 18:11:11.364558 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.565078    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-2c9b1d: configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.364773 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.565125    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume podName:3d96e066-10c0-4fe2-9e24-ffd6195b22af nodeName:}" failed. No retries permitted until 2026-11-02 18:11:08.065109035 +0000 UTC m=+2484.556172188 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" (UID: "3d96e066-10c0-4fe2-9e24-ffd6195b22af") : configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.364773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565161    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client-kube-rbac-proxy-config\""
Nov 02 18:11:11.364773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565205    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"kuadrant-console-nginx-conf\""
Nov 02 18:11:11.364773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565227    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-distinct-2-simulated-kserve-self-signed-certs\""
Nov 02 18:11:11.364773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565252    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.364773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565290    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.364773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565339    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.364773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565342    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"simulator-w5-863b87-kserve-self-signed-certs\""
Nov 02 18:11:11.364773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565370    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-web\""
Nov 02 18:11:11.365144 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565380    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.365144 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565413    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"authorino-server-cert\""
Nov 02 18:11:11.365144 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.565432    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-96768b: configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.365144 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565462    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-trlp-test-simulated-kserve-self-signed-certs\""
Nov 02 18:11:11.365144 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.565471    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume podName:22b556c1-112b-49d4-932c-0143e912a54e nodeName:}" failed. No retries permitted until 2026-11-02 18:11:08.065460717 +0000 UTC m=+2484.556523855 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" (UID: "22b556c1-112b-49d4-932c-0143e912a54e") : configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.365144 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565549    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 18:11:11.365144 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565565    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"plugin-serving-cert\""
Nov 02 18:11:11.365144 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565574    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"distinct-w3-5eedab-kserve-self-signed-certs\""
Nov 02 18:11:11.365144 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565587    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-kube-rbac-proxy-config\""
Nov 02 18:11:11.365483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565616    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"distinct-2-w1-120394-kserve-self-signed-certs\""
Nov 02 18:11:11.365483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565652    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-b-w3-227939\""
Nov 02 18:11:11.365483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565661    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"federate-client-certs\""
Nov 02 18:11:11.365483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565552    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.365483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565709    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.365483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.565260    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.365483 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.567549    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-96768b: configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.365483 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.567593    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume podName:5ce3335c-36e6-4018-a264-5b79085f09f3 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:08.067580607 +0000 UTC m=+2484.558643746 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume") pod "payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" (UID: "5ce3335c-36e6-4018-a264-5b79085f09f3") : configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.365483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.568720    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"openshift-state-metrics-tls\""
Nov 02 18:11:11.365858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.568753    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-client-serving-certs-ca-bundle\""
Nov 02 18:11:11.365858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.568952    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy\""
Nov 02 18:11:11.365858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.568973    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w2-f0a3b8\""
Nov 02 18:11:11.365858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.569028    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.365858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.569065    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.365858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.569097    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"original-pull-secret\""
Nov 02 18:11:11.365858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.569098    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-grpc-tls-5oaelllrcvii0\""
Nov 02 18:11:11.365858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.569107    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-a-w5-504de7-serving-cert\""
Nov 02 18:11:11.365858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.569139    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"premium-w3-5eedab-kserve-self-signed-certs\""
Nov 02 18:11:11.365858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.569965    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-a29999\""
Nov 02 18:11:11.365858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.574156    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-w2-f0a3b8-serving-cert\""
Nov 02 18:11:11.365858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.575836    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-client-ca\""
Nov 02 18:11:11.366375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.577232    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"telemeter-trusted-ca-bundle-28r105lbnph2c\""
Nov 02 18:11:11.366375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.577393    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-script-lib\""
Nov 02 18:11:11.366375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.577442    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-insights\"/\"insights-runtime-extractor-tls\""
Nov 02 18:11:11.366375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.577479    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress-canary\"/\"canary-serving-cert\""
Nov 02 18:11:11.366375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.577534    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.366375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.577540    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"env-overrides\""
Nov 02 18:11:11.366375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.577574    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-inf-b-d5b905-dockercfg-sfklt\""
Nov 02 18:11:11.366375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.577614    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-tls\""
Nov 02 18:11:11.366375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.577673    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"ovnkube-config\""
Nov 02 18:11:11.366375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.577684    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"metrics-server-cert\""
Nov 02 18:11:11.366375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.577707    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"kube-rbac-proxy\""
Nov 02 18:11:11.366375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.577715    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-jobset-operator\"/\"webhook-server-cert\""
Nov 02 18:11:11.366878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.577732    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ovn-kubernetes\"/\"ovn-node-metrics-cert\""
Nov 02 18:11:11.366878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.578005    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.366878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.578035    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"premium-w1-120394-kserve-self-signed-certs\""
Nov 02 18:11:11.366878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.578050    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"telemeter-client\""
Nov 02 18:11:11.366878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.578074    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.366878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.578110    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-cluster-csi-drivers\"/\"aws-ebs-csi-driver-node-sa-dockercfg-t2hrp\""
Nov 02 18:11:11.366878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.578141    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-storage-version-migrator-sa-dockercfg-vqf9q\""
Nov 02 18:11:11.366878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.578151    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"thanos-querier-kube-rbac-proxy-metrics\""
Nov 02 18:11:11.366878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.578181    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.366878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.578190    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"cert-manager\"/\"cert-manager-dockercfg-84fvd\""
Nov 02 18:11:11.366878 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.578923    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a29999: configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.367282 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.578978    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume podName:7e09749e-e648-40a3-a5f1-f0dc35fccf24 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:08.078961446 +0000 UTC m=+2484.570024604 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" (UID: "7e09749e-e648-40a3-a5f1-f0dc35fccf24") : configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.367282 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.579040    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a29999: configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.367282 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.579096    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume podName:1c190514-ba33-4844-a9ec-e4bb47b5f706 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:08.079080728 +0000 UTC m=+2484.570143890 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume") pod "payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" (UID: "1c190514-ba33-4844-a9ec-e4bb47b5f706") : configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.367282 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.579756    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367282 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.579796    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"llm\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.579839    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="csi-driver" containerStatusID={"Type":"cri-o","ID":"ad864ff6740e4c778f72c8079e4f03eacc7840d208ff11512bdc1dc1655e8c7a"} pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" containerMessage="Container csi-driver failed liveness probe, will be restarted"
Nov 02 18:11:11.367509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.579870    4456 kuberuntime_manager.go:1329] "Message for Container of pod" containerName="csi-node-driver-registrar" containerStatusID={"Type":"cri-o","ID":"2f2d0ecef34f90d7be65ae3bfde69eebfd2e056a726c73ec5c4cc23ac48cb253"} pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" containerMessage="Container csi-node-driver-registrar failed liveness probe, will be restarted"
Nov 02 18:11:11.367509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.579920    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-driver" containerID="cri-o://ad864ff6740e4c778f72c8079e4f03eacc7840d208ff11512bdc1dc1655e8c7a" gracePeriod=30
Nov 02 18:11:11.367509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.580259    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kuadrant-system\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.580318    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ovn-kubernetes\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.580597    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.585008    4456 kubelet_pods.go:1082] "Unable to retrieve pull secret, the image pull may not succeed." pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" secret="" err="secret \"payload-processing-e2e-inf-b-d5b905-dockercfg-sfklt\" not found"
Nov 02 18:11:11.367509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.585305    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.585483    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress-canary\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.585550    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"cert-manager\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.585936    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"opendatahub-operator-system\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.586101    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.588627    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-diagnostics\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.592979    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-insights\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.593155    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-shared-a-w5-504de7-dockercfg-d4zk4\""
Nov 02 18:11:11.367845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.593373    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.593767    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.593951    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-console\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.617309    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-jobset-operator\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.367845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.617427    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.368370 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.618287    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-b-w5-c810ca-openshift-default-dockercfg-s9n8v\""
Nov 02 18:11:11.368370 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.682056    4456 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientMemory"
Nov 02 18:11:11.368370 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.682093    4456 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasNoDiskPressure"
Nov 02 18:11:11.368370 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.682109    4456 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeHasSufficientPID"
Nov 02 18:11:11.368370 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.682130    4456 kubelet_node_status.go:683] "Recording event message for node" node="ip-10-0-141-87.ec2.internal" event="NodeReady"
Nov 02 18:11:11.368370 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.682890    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:11.368370 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.736572    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"whereabouts-flatfile-config\""
Nov 02 18:11:11.368370 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.737178    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-image-registry\"/\"trusted-ca\""
Nov 02 18:11:11.368370 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.737283    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-8a209f\""
Nov 02 18:11:11.368370 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.737366    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-kube-rbac-proxy-config\""
Nov 02 18:11:11.368370 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.794122    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-872354: configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.368772 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.794198    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume podName:44b66f41-99ae-446b-a50a-2ec0b71563a4 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:08.294178607 +0000 UTC m=+2484.785241771 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" (UID: "44b66f41-99ae-446b-a50a-2ec0b71563a4") : configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.368772 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.817549    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a5c1da: configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.368772 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.817637    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume podName:8c60dfaa-dabe-4726-91b5-165443ff3971 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:08.317617833 +0000 UTC m=+2484.808680994 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume") pod "payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" (UID: "8c60dfaa-dabe-4726-91b5-165443ff3971") : configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.368772 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.843131    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-2c9b1d: configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.368998 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:07.843225    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume podName:60c0e838-5c79-4bf7-a7c6-87903ae8bd8c nodeName:}" failed. No retries permitted until 2026-11-02 18:11:08.343195323 +0000 UTC m=+2484.834258484 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume") pod "payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" (UID: "60c0e838-5c79-4bf7-a7c6-87903ae8bd8c") : configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.368998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.895447    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 18:11:11.368998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.898159    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w3-5eedab\"/\"simulator-w3-5eedab-kserve-self-signed-certs\""
Nov 02 18:11:11.368998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.932800    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:11.368998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:07.954692    4456 generic.go:356] "Generic (PLEG): container finished" podID="18d2ae0a-bf65-4aee-8730-093e00b54a24" containerID="c426b7d4950263c913e37548ad7992f0212cc9969a4598d6b4fc3ff560c31e70" exitCode=0
Nov 02 18:11:11.368998 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:07.960797    4456 logging.go:55] [core] [Channel #6641 SubChannel #6667]grpc: addrConn.createTransport failed to connect to {Addr: "/var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock", ServerName: "localhost", BalancerAttributes: {"<%!p(pickfirstleaf.managedByPickfirstKeyType={})>": "<%!p(bool=true)>" }}. Err: connection error: desc = "error reading server preface: read unix @->/csi/csi.sock: read: connection reset by peer"
Nov 02 18:11:11.369259 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.211599    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-prometheus-http-client-file\""
Nov 02 18:11:11.369259 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.216789    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-client-certs\""
Nov 02 18:11:11.369259 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.217131    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-96768b: configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.369259 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.217215    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume podName:5ce3335c-36e6-4018-a264-5b79085f09f3 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:09.217190468 +0000 UTC m=+2485.708253613 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume") pod "payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" (UID: "5ce3335c-36e6-4018-a264-5b79085f09f3") : configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.369259 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.217222    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-2c9b1d: configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.369259 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.217277    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-96768b: configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.369507 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.217307    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume podName:3d96e066-10c0-4fe2-9e24-ffd6195b22af nodeName:}" failed. No retries permitted until 2026-11-02 18:11:09.217288115 +0000 UTC m=+2485.708351281 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" (UID: "3d96e066-10c0-4fe2-9e24-ffd6195b22af") : configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.369507 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.217641    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume podName:22b556c1-112b-49d4-932c-0143e912a54e nodeName:}" failed. No retries permitted until 2026-11-02 18:11:09.217625022 +0000 UTC m=+2485.708688165 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" (UID: "22b556c1-112b-49d4-932c-0143e912a54e") : configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.369507 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.217647    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a29999: configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.369647 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.217707    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume podName:1c190514-ba33-4844-a9ec-e4bb47b5f706 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:09.217681732 +0000 UTC m=+2485.708744878 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume") pod "payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" (UID: "1c190514-ba33-4844-a9ec-e4bb47b5f706") : configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.369647 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.217730    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a29999: configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.369647 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.217781    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume podName:7e09749e-e648-40a3-a5f1-f0dc35fccf24 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:09.217769298 +0000 UTC m=+2485.708832462 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" (UID: "7e09749e-e648-40a3-a5f1-f0dc35fccf24") : configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.369647 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.218054    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-872354: configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.369847 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.218102    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume podName:cf85b137-2b1e-4e22-8f6e-744a02ea4e46 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:09.218089921 +0000 UTC m=+2485.709153079 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume") pod "payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" (UID: "cf85b137-2b1e-4e22-8f6e-744a02ea4e46") : configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.369847 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.218162    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a5c1da: configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.369847 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.218199    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume podName:96db7b02-b878-435a-869e-1d30e147c61d nodeName:}" failed. No retries permitted until 2026-11-02 18:11:09.218189581 +0000 UTC m=+2485.709252764 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" (UID: "96db7b02-b878-435a-869e-1d30e147c61d") : configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.369847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.218512    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-kube-rbac-proxy-web\""
Nov 02 18:11:11.369847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.238634    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:11.370060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.276121    4456 csi_client.go:182] "Error calling CSI NodeGetInfo()" err="rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: read unix @->/csi/csi.sock: read: connection reset by peer\""
Nov 02 18:11:11.370060 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.276170    4456 csi_node_updater.go:178] "Failed to update CSIDriver" err="failed to get NodeGetInfo from driver \"ebs.csi.aws.com\": rpc error: code = Unavailable desc = connection error: desc = \"error reading server preface: read unix @->/csi/csi.sock: read: connection reset by peer\"" driver="ebs.csi.aws.com"
Nov 02 18:11:11.370060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.320837    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"iptables-alerter-script\""
Nov 02 18:11:11.370060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.321223    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w5-863b87\"/\"distinct-2-w5-863b87-kserve-self-signed-certs\""
Nov 02 18:11:11.370060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.321807    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-d5b905\""
Nov 02 18:11:11.370060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.321933    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-multus\"/\"default-cni-sysctl-allowlist\""
Nov 02 18:11:11.370060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.322098    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.370060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.322195    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 18:11:11.370060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.322292    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-b-fbbbf8\""
Nov 02 18:11:11.370060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.322480    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls-assets-0\""
Nov 02 18:11:11.370450 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.322731    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 18:11:11.370450 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.322862    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-fbbbf8: configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.370450 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.322951    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume podName:85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:08.822926827 +0000 UTC m=+2485.313989989 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume") pod "payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" (UID: "85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3") : configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.370450 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.323162    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"kube-rbac-proxy\""
Nov 02 18:11:11.370450 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.323317    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.370450 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.323324    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-grpc-tls-5mdnhf4neu495\""
Nov 02 18:11:11.370450 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.323382    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-1\""
Nov 02 18:11:11.370450 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.323426    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-2\""
Nov 02 18:11:11.370450 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.323451    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"node-exporter-tls\""
Nov 02 18:11:11.370797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.323322    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-web-config\""
Nov 02 18:11:11.370797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.323474    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-w4-2899ab\""
Nov 02 18:11:11.370797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.323562    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-operator-admission-webhook-tls\""
Nov 02 18:11:11.370797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.323675    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-shared-a-w5-504de7\""
Nov 02 18:11:11.370797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.323798    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w1-120394\"/\"simulator-w1-120394-kserve-self-signed-certs\""
Nov 02 18:11:11.370797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.323910    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-inf-a-bb6d2e\""
Nov 02 18:11:11.370797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.323915    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"kubelet-serving-ca-bundle\""
Nov 02 18:11:11.370797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.324011    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.370797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.323941    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-private-configuration\""
Nov 02 18:11:11.370797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.324074    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.370797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.324149    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent-dockercfg-mwsx7\""
Nov 02 18:11:11.370797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.324188    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-dns\"/\"dns-default\""
Nov 02 18:11:11.371293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.324276    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"serving-certs-ca-bundle\""
Nov 02 18:11:11.371293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.324322    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"image-registry-tls\""
Nov 02 18:11:11.371293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.324332    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s\""
Nov 02 18:11:11.371293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.324391    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-thanos-sidecar-tls\""
Nov 02 18:11:11.371293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.324450    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-multus\"/\"metrics-daemon-secret\""
Nov 02 18:11:11.371293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.324576    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"kube-root-ca.crt\""
Nov 02 18:11:11.371293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.324624    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"node-exporter-accelerators-collector-config\""
Nov 02 18:11:11.371293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.324674    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-tls\""
Nov 02 18:11:11.371293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.324728    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"premium-w0-069fa5-kserve-self-signed-certs\""
Nov 02 18:11:11.371293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.324852    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-image-registry\"/\"installation-pull-secrets\""
Nov 02 18:11:11.371293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.324970    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-dns\"/\"dns-default-metrics-tls\""
Nov 02 18:11:11.371293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.325053    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-shared-b-w4-2672ed-serving-cert\""
Nov 02 18:11:11.371767 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.325840    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-trusted-ca-bundle\""
Nov 02 18:11:11.371767 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.328396    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"kube-auth-proxy-tls\""
Nov 02 18:11:11.371767 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.333357    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-872354: configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.371767 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.333433    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume podName:44b66f41-99ae-446b-a50a-2ec0b71563a4 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:09.333412031 +0000 UTC m=+2485.824475173 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" (UID: "44b66f41-99ae-446b-a50a-2ec0b71563a4") : configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.371767 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.333496    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a5c1da: configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.372010 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.333537    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume podName:8c60dfaa-dabe-4726-91b5-165443ff3971 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:09.333528228 +0000 UTC m=+2485.824591370 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume") pod "payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" (UID: "8c60dfaa-dabe-4726-91b5-165443ff3971") : configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.372010 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.337199    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-d5b905: configmap "payload-processing-plugins-e2e-inf-b-d5b905" not found
Nov 02 18:11:11.372010 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.337260    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume podName:e141d845-db9d-4f94-b05e-cfb7848e8fef nodeName:}" failed. No retries permitted until 2026-11-02 18:11:08.837235734 +0000 UTC m=+2485.328298879 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume") pod "payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" (UID: "e141d845-db9d-4f94-b05e-cfb7848e8fef") : configmap "payload-processing-plugins-e2e-inf-b-d5b905" not found
Nov 02 18:11:11.372010 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.338212    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-fbbbf8: configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.372157 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.338285    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume podName:3d398f17-735a-4d4b-b1cc-5b24fd266535 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:08.838259667 +0000 UTC m=+2485.329322809 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" (UID: "3d398f17-735a-4d4b-b1cc-5b24fd266535") : configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.372157 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.339031    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-bb6d2e: configmap "payload-processing-plugins-e2e-inf-a-bb6d2e" not found
Nov 02 18:11:11.372157 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.339089    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume podName:f4ee5c50-5738-496a-b5ef-c86ead76e0f6 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:08.839074235 +0000 UTC m=+2485.330137377 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume") pod "payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" (UID: "f4ee5c50-5738-496a-b5ef-c86ead76e0f6") : configmap "payload-processing-plugins-e2e-inf-a-bb6d2e" not found
Nov 02 18:11:11.372157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.340433    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-node-tuning-operator\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.372157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.341164    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"prometheus-k8s-rulefiles-0\""
Nov 02 18:11:11.372375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.342466    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-network-operator\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.372375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.348971    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kube-system\"/\"konnectivity-agent\""
Nov 02 18:11:11.372375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.349093    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"kube-system\"/\"konnectivity-ca-bundle\""
Nov 02 18:11:11.372375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.355228    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-cluster-csi-drivers\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.372375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.355499    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"e2e-models-e2e-worker-w0-069fa5\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.372375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.355569    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-kube-storage-version-migrator\"/\"openshift-service-ca.crt\""
Nov 02 18:11:11.372375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.384899    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" podUID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.20:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.372375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.396031    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" probeResult="failure" output="Get \"http://10.134.0.27:8081/readyz\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:11:11.372375 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.440775    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-2c9b1d: configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.372634 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.440878    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume podName:60c0e838-5c79-4bf7-a7c6-87903ae8bd8c nodeName:}" failed. No retries permitted until 2026-11-02 18:11:09.440854483 +0000 UTC m=+2485.931917651 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume") pod "payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" (UID: "60c0e838-5c79-4bf7-a7c6-87903ae8bd8c") : configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.372634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.480581    4456 prober.go:122] "Probe failed" probeType="Liveness" pod="cert-manager/cert-manager-597b95cb84-hr8vz" podUID="e73be583-2a26-4379-8169-0a6b8784a11b" containerName="cert-manager-controller" probeResult="failure" output="Get \"http://10.134.0.26:9403/livez\": dial tcp 10.134.0.26:9403: connect: connection refused"
Nov 02 18:11:11.372634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.577209    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fae57b7d-d41c-4b47-a4c6-6ff96e936ddb" path="/var/lib/kubelet/pods/fae57b7d-d41c-4b47-a4c6-6ff96e936ddb/volumes"
Nov 02 18:11:11.372634 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.856444    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-d5b905: configmap "payload-processing-plugins-e2e-inf-b-d5b905" not found
Nov 02 18:11:11.372771 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.856515    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume podName:e141d845-db9d-4f94-b05e-cfb7848e8fef nodeName:}" failed. No retries permitted until 2026-11-02 18:11:09.856497913 +0000 UTC m=+2486.347561072 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume") pod "payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" (UID: "e141d845-db9d-4f94-b05e-cfb7848e8fef") : configmap "payload-processing-plugins-e2e-inf-b-d5b905" not found
Nov 02 18:11:11.372771 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.856560    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-fbbbf8: configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.372771 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.856596    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume podName:85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:09.856584173 +0000 UTC m=+2486.347647325 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume") pod "payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" (UID: "85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3") : configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.372771 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.856636    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-bb6d2e: configmap "payload-processing-plugins-e2e-inf-a-bb6d2e" not found
Nov 02 18:11:11.372928 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.856667    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume podName:f4ee5c50-5738-496a-b5ef-c86ead76e0f6 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:09.856657284 +0000 UTC m=+2486.347720440 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume") pod "payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" (UID: "f4ee5c50-5738-496a-b5ef-c86ead76e0f6") : configmap "payload-processing-plugins-e2e-inf-a-bb6d2e" not found
Nov 02 18:11:11.372928 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.856702    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-fbbbf8: configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.372928 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.856733    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume podName:3d398f17-735a-4d4b-b1cc-5b24fd266535 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:09.856723175 +0000 UTC m=+2486.347786327 (durationBeforeRetry 1s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" (UID: "3d398f17-735a-4d4b-b1cc-5b24fd266535") : configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.372928 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:08.882986    4456 kubelet.go:2711] "Housekeeping took longer than expected" err="housekeeping took too long" expected="1s" actual="1.541s"
Nov 02 18:11:11.372928 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.883049    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 18:11:11.372928 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.936951    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 18:11:11.373187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.937051    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 18:11:11.373187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.937070    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d987f75-ntp98" event={"ID":"18d2ae0a-bf65-4aee-8730-093e00b54a24","Type":"ContainerDied","Data":"c426b7d4950263c913e37548ad7992f0212cc9969a4598d6b4fc3ff560c31e70"}
Nov 02 18:11:11.373187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.937102    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 18:11:11.373187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.937126    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 18:11:11.373187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.937146    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 18:11:11.373187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.937171    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"]
Nov 02 18:11:11.373187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.938664    4456 cpu_manager.go:414] "containerMap: removing container" podUID="bb80eb43-a6a5-4388-8436-6618c2f1493d" containerName="revoke-keys"
Nov 02 18:11:11.373187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.938691    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bb80eb43-a6a5-4388-8436-6618c2f1493d" containerName="revoke-keys"
Nov 02 18:11:11.373187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:08.958604    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="bb80eb43-a6a5-4388-8436-6618c2f1493d" containerName="revoke-keys"
Nov 02 18:11:11.373187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:09.082194    4456 generic.go:356] "Generic (PLEG): container finished" podID="9fe1660e-3733-4c7f-bc5c-3820d84aa143" containerID="cba9f82f9983882e3976ebd947a2c9e62cc786d2b9bba3e2b09311bc77e055cb" exitCode=1
Nov 02 18:11:11.373187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:09.082434    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.373764 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:09.083665    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.373764 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.277714    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-96768b: configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.373764 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.277805    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a29999: configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.373764 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.277827    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a29999: configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.373764 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.277877    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume podName:1c190514-ba33-4844-a9ec-e4bb47b5f706 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:11.277856539 +0000 UTC m=+2487.768919678 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume") pod "payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" (UID: "1c190514-ba33-4844-a9ec-e4bb47b5f706") : configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.373996 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.277904    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume podName:5ce3335c-36e6-4018-a264-5b79085f09f3 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:11.277894818 +0000 UTC m=+2487.768957963 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume") pod "payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" (UID: "5ce3335c-36e6-4018-a264-5b79085f09f3") : configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.373996 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.277925    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume podName:7e09749e-e648-40a3-a5f1-f0dc35fccf24 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:11.27791621 +0000 UTC m=+2487.768979352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" (UID: "7e09749e-e648-40a3-a5f1-f0dc35fccf24") : configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.373996 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.278225    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-872354: configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.374121 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.278269    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume podName:cf85b137-2b1e-4e22-8f6e-744a02ea4e46 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:11.278255924 +0000 UTC m=+2487.769319078 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume") pod "payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" (UID: "cf85b137-2b1e-4e22-8f6e-744a02ea4e46") : configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.374121 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.278299    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-2c9b1d: configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.374121 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.278334    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume podName:3d96e066-10c0-4fe2-9e24-ffd6195b22af nodeName:}" failed. No retries permitted until 2026-11-02 18:11:11.278323992 +0000 UTC m=+2487.769387136 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" (UID: "3d96e066-10c0-4fe2-9e24-ffd6195b22af") : configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.374121 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.278369    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-96768b: configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.374297 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.278403    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume podName:22b556c1-112b-49d4-932c-0143e912a54e nodeName:}" failed. No retries permitted until 2026-11-02 18:11:11.278392595 +0000 UTC m=+2487.769455749 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" (UID: "22b556c1-112b-49d4-932c-0143e912a54e") : configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.374297 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.278421    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a5c1da: configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.374297 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.278451    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume podName:96db7b02-b878-435a-869e-1d30e147c61d nodeName:}" failed. No retries permitted until 2026-11-02 18:11:11.278442114 +0000 UTC m=+2487.769505256 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" (UID: "96db7b02-b878-435a-869e-1d30e147c61d") : configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.374297 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.381568    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a5c1da: configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.374477 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.381634    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume podName:8c60dfaa-dabe-4726-91b5-165443ff3971 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:11.381615006 +0000 UTC m=+2487.872678151 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume") pod "payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" (UID: "8c60dfaa-dabe-4726-91b5-165443ff3971") : configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.374477 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.381678    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-872354: configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.374477 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.381712    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume podName:44b66f41-99ae-446b-a50a-2ec0b71563a4 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:11.381701953 +0000 UTC m=+2487.872765095 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" (UID: "44b66f41-99ae-446b-a50a-2ec0b71563a4") : configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.374477 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:09.410128    4456 trace.go:236] Trace[2050819939]: "Calculate volume metrics of manifests for pod opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" (02-Nov-2026 18:07:41.839) (total time: 207570ms):
Nov 02 18:11:11.374477 ip-10-0-141-87 kubenswrapper[4456]: Trace[2050819939]: [3m27.570690134s] [3m27.570690134s] END
Nov 02 18:11:11.374720 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:09.457675    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:11.374720 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.491221    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-2c9b1d: configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.374720 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.491322    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume podName:60c0e838-5c79-4bf7-a7c6-87903ae8bd8c nodeName:}" failed. No retries permitted until 2026-11-02 18:11:11.491297006 +0000 UTC m=+2487.982360191 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume") pod "payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" (UID: "60c0e838-5c79-4bf7-a7c6-87903ae8bd8c") : configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.374720 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:09.851409    4456 patch_prober.go:29] interesting pod/thanos-querier-667db95b96-kmwrg container/kube-rbac-proxy-web namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.15:9091/-/ready\": dial tcp 10.134.0.15:9091: connect: connection refused" start-of-body=
Nov 02 18:11:11.374720 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:09.851470    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" podUID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerName="kube-rbac-proxy-web" probeResult="failure" output="Get \"https://10.134.0.15:9091/-/ready\": dial tcp 10.134.0.15:9091: connect: connection refused"
Nov 02 18:11:11.374720 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.872891    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-fbbbf8: configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.374986 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.872967    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume podName:3d398f17-735a-4d4b-b1cc-5b24fd266535 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:11.872946785 +0000 UTC m=+2488.364009933 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" (UID: "3d398f17-735a-4d4b-b1cc-5b24fd266535") : configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.374986 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.873314    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-d5b905: configmap "payload-processing-plugins-e2e-inf-b-d5b905" not found
Nov 02 18:11:11.374986 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.873370    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume podName:e141d845-db9d-4f94-b05e-cfb7848e8fef nodeName:}" failed. No retries permitted until 2026-11-02 18:11:11.873353813 +0000 UTC m=+2488.364416958 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume") pod "payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" (UID: "e141d845-db9d-4f94-b05e-cfb7848e8fef") : configmap "payload-processing-plugins-e2e-inf-b-d5b905" not found
Nov 02 18:11:11.374986 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.873412    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-fbbbf8: configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.375148 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.873443    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume podName:85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:11.873433108 +0000 UTC m=+2488.364496246 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume") pod "payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" (UID: "85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3") : configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.375148 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.873479    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-bb6d2e: configmap "payload-processing-plugins-e2e-inf-a-bb6d2e" not found
Nov 02 18:11:11.375148 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:09.873511    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume podName:f4ee5c50-5738-496a-b5ef-c86ead76e0f6 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:11.873500196 +0000 UTC m=+2488.364563352 (durationBeforeRetry 2s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume") pod "payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" (UID: "f4ee5c50-5738-496a-b5ef-c86ead76e0f6") : configmap "payload-processing-plugins-e2e-inf-a-bb6d2e" not found
Nov 02 18:11:11.375148 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.104127    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"
Nov 02 18:11:11.375148 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.122163    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:11:11.375350 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.122226    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:11:11.375350 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.322901    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3515275e-be1d-4ebe-b7fb-41178d545638-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr\" (UID: \"3515275e-be1d-4ebe-b7fb-41178d545638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"
Nov 02 18:11:11.375350 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.323315    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v787t\" (UniqueName: \"kubernetes.io/projected/3515275e-be1d-4ebe-b7fb-41178d545638-kube-api-access-v787t\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr\" (UID: \"3515275e-be1d-4ebe-b7fb-41178d545638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"
Nov 02 18:11:11.375350 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.323613    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3515275e-be1d-4ebe-b7fb-41178d545638-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr\" (UID: \"3515275e-be1d-4ebe-b7fb-41178d545638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"
Nov 02 18:11:11.375350 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.323606    4456 generic.go:356] "Generic (PLEG): container finished" podID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerID="4cee89e694b5799548d6085cbcaa2612948ac905b236f27ac06a462827cad81c" exitCode=0
Nov 02 18:11:11.375350 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.422134    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 18:11:11.375350 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.422195    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-80911fb3-xrwb7"]
Nov 02 18:11:11.375634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.422222    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 18:11:11.375634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.422243    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 18:11:11.375634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.422264    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9" event={"ID":"9fe1660e-3733-4c7f-bc5c-3820d84aa143","Type":"ContainerDied","Data":"cba9f82f9983882e3976ebd947a2c9e62cc786d2b9bba3e2b09311bc77e055cb"}
Nov 02 18:11:11.375634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.422298    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-pgmzg" event={"ID":"6e78d6c7-c325-48f2-af90-804dade84d8f","Type":"ContainerDied","Data":"4cee89e694b5799548d6085cbcaa2612948ac905b236f27ac06a462827cad81c"}
Nov 02 18:11:11.375634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.422321    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-v2zdq","odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-zh652","odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-tchlj","odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vhqq9"]
Nov 02 18:11:11.375634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.428331    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3515275e-be1d-4ebe-b7fb-41178d545638-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr\" (UID: \"3515275e-be1d-4ebe-b7fb-41178d545638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"
Nov 02 18:11:11.375988 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.428684    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v787t\" (UniqueName: \"kubernetes.io/projected/3515275e-be1d-4ebe-b7fb-41178d545638-kube-api-access-v787t\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr\" (UID: \"3515275e-be1d-4ebe-b7fb-41178d545638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"
Nov 02 18:11:11.375988 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.429080    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3515275e-be1d-4ebe-b7fb-41178d545638-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr\" (UID: \"3515275e-be1d-4ebe-b7fb-41178d545638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"
Nov 02 18:11:11.375988 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.455978    4456 scope.go:122] "RemoveContainer" containerID="cba9f82f9983882e3976ebd947a2c9e62cc786d2b9bba3e2b09311bc77e055cb"
Nov 02 18:11:11.376126 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:10.481309    4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec905c34_6dfb_483f_bb3d_7ba412262584.slice/crio-conmon-a3e5b342fbd24d60a087ce776f7958683a916f3ab12b9d0f407b62ce53f0987a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73be583_2a26_4379_8169_0a6b8784a11b.slice/crio-conmon-d381788d1f97c945f12e6a0f9ca8aea3a3f0d92249a339bd95400d14c394ce40.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebd3467_841a_4331_b149_8acf6e60c12c.slice/crio-conmon-e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2d24bd_15b2_4eef_a5fc_5cfc3be9417f.slice/crio-conmon-b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:11:11.376126 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:10.482437    4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2d24bd_15b2_4eef_a5fc_5cfc3be9417f.slice/crio-conmon-b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebd3467_841a_4331_b149_8acf6e60c12c.slice/crio-conmon-e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pode73be583_2a26_4379_8169_0a6b8784a11b.slice/crio-conmon-d381788d1f97c945f12e6a0f9ca8aea3a3f0d92249a339bd95400d14c394ce40.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:11:11.376232 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:10.483351    4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebd3467_841a_4331_b149_8acf6e60c12c.slice/crio-conmon-e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2d24bd_15b2_4eef_a5fc_5cfc3be9417f.slice/crio-conmon-b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:11:11.376232 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:10.494787    4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfb44e7b7_e0cb_4270_b14c_9c2397f37269.slice/crio-conmon-61f0d172af153366e5fe0177879402cf8b046838c4d8800dd144c8d0095dd565.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podec905c34_6dfb_483f_bb3d_7ba412262584.slice/crio-conmon-a3e5b342fbd24d60a087ce776f7958683a916f3ab12b9d0f407b62ce53f0987a.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6ebd3467_841a_4331_b149_8acf6e60c12c.slice/crio-conmon-e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2d24bd_15b2_4eef_a5fc_5cfc3be9417f.slice/crio-conmon-b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436.scope\": RecentStats: unable to find data in memory cache], [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ea3dac_7073_4564_b247_89ea94a64342.slice/crio-conmon-f46a45f18a610493bf46d8787ee563938a877ab2e179a2e516f44f6e70fcc573.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:11:11.376348 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:10.517265    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:11:11.376348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.540020    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-zh652"
Nov 02 18:11:11.376348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.589213    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3515275e-be1d-4ebe-b7fb-41178d545638-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr\" (UID: \"3515275e-be1d-4ebe-b7fb-41178d545638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"
Nov 02 18:11:11.376348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.626361    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-zh652"
Nov 02 18:11:11.376348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.627283    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-v2zdq"
Nov 02 18:11:11.376348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.642392    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-tchlj"
Nov 02 18:11:11.376348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.655440    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 18:11:11.376348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.655559    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 18:11:11.376348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.655583    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"]
Nov 02 18:11:11.376348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.655642    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 18:11:11.376734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.655939    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vhqq9"]
Nov 02 18:11:11.376734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656059    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 18:11:11.376734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656091    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2"]
Nov 02 18:11:11.376734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656115    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-console/downloads-6dd789c8b7-pgmzg"
Nov 02 18:11:11.376734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656152    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 18:11:11.376734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656176    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/downloads-6dd789c8b7-pgmzg"]
Nov 02 18:11:11.376734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656261    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 18:11:11.376734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656304    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"
Nov 02 18:11:11.376734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656348    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 18:11:11.376734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656365    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 18:11:11.376734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656385    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"]
Nov 02 18:11:11.376734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656411    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"]
Nov 02 18:11:11.377219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656441    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 18:11:11.377219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656488    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 18:11:11.377219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656524    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 18:11:11.377219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656544    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 18:11:11.377219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656589    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 18:11:11.377219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656629    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 18:11:11.377219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656647    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n"
Nov 02 18:11:11.377219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656685    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 18:11:11.377219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656715    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 18:11:11.377219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656731    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"]
Nov 02 18:11:11.377219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.656997    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"]
Nov 02 18:11:11.377219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657081    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"]
Nov 02 18:11:11.377693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657112    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"]
Nov 02 18:11:11.377693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657169    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 18:11:11.377693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657206    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 18:11:11.377693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657219    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"]
Nov 02 18:11:11.377693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657249    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:11:11.377693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657281    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 18:11:11.377693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657294    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"]
Nov 02 18:11:11.377693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657319    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"]
Nov 02 18:11:11.377693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657344    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"]
Nov 02 18:11:11.377693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657369    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"]
Nov 02 18:11:11.377693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657395    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"]
Nov 02 18:11:11.377693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657424    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-trlp-test-simulated-kserve-69b588897d-xc9kg"
Nov 02 18:11:11.378189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657444    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"]
Nov 02 18:11:11.378189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657486    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 18:11:11.378189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657516    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 18:11:11.378189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657529    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"]
Nov 02 18:11:11.378189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657556    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 18:11:11.378189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657584    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 18:11:11.378189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657596    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"]
Nov 02 18:11:11.378189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657640    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k"
Nov 02 18:11:11.378189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657659    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"]
Nov 02 18:11:11.378189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657678    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 18:11:11.378189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657704    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-07748554-wnr4k"]
Nov 02 18:11:11.378189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657726    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"]
Nov 02 18:11:11.378662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657756    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 18:11:11.378662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.657801    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 18:11:11.378662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.659592    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"]
Nov 02 18:11:11.378662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.659672    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 18:11:11.378662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.659695    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"]
Nov 02 18:11:11.378662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.659719    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"]
Nov 02 18:11:11.378662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.659751    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 18:11:11.378662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.659770    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"]
Nov 02 18:11:11.378662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.659796    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"]
Nov 02 18:11:11.378662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.659945    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 18:11:11.378662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.660017    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 18:11:11.378662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.660053    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 18:11:11.379182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.660076    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 18:11:11.379182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.660115    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 18:11:11.379182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.660153    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 18:11:11.379182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.660183    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:11:11.379182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.660215    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 18:11:11.379182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.660242    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 18:11:11.379182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.660266    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 18:11:11.379182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.660291    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 18:11:11.379182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.660321    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:11:11.379182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.660344    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 18:11:11.379182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.660376    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 18:11:11.379972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.660400    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-console/downloads-6dd789c8b7-pgmzg"
Nov 02 18:11:11.379972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.661641    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-dns/dns-default-zmrdt"
Nov 02 18:11:11.379972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.661690    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 18:11:11.379972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.661712    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 18:11:11.379972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.661728    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 18:11:11.379972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.661745    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 18:11:11.379972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.661762    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 18:11:11.379972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.661835    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"
Nov 02 18:11:11.379972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.661876    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 18:11:11.379972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.661936    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 18:11:11.379972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.661964    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"
Nov 02 18:11:11.380388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.661988    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"
Nov 02 18:11:11.380388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.662061    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 18:11:11.380388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.662092    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:11:11.380388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.662118    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 18:11:11.380388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.662141    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 18:11:11.380388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.663524    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 18:11:11.380388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.663559    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 18:11:11.380388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.663585    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"
Nov 02 18:11:11.380388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.663608    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"
Nov 02 18:11:11.380388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.665211    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 18:11:11.380388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.665244    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 18:11:11.380829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.665296    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"
Nov 02 18:11:11.380829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.665339    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 18:11:11.380829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.665473    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.380829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.667761    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vhqq9"
Nov 02 18:11:11.380829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.672901    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.380829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.674566    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.380829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.674683    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy" containerID="cri-o://7b5ffd17510b581a401cae0b3017c71076c1341b3ae93eed0e79ed9040740ec5" gracePeriod=30
Nov 02 18:11:11.380829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.680482    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.380829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.680567    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy" containerID="cri-o://cae3f46a8bc87e7369f5ef014d19a5821ecaa21b04f97dd5ed71d6ee6eb99acf" gracePeriod=30
Nov 02 18:11:11.380829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.682288    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.380829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.683031    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.380829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.683155    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.390228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.687721    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.390228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.687921    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy" containerID="cri-o://e0bbfd6f9c197b1322c631c19d97f927c7c4b56631d9cd54e8dd5474d3b87b5d" gracePeriod=30
Nov 02 18:11:11.390228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.688121    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.390228 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:10.690712    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="unmounted volumes=[kube-api-access-v787t maas-api-xks-ca], unattached volumes=[], failed to process volumes=[]: context canceled" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr" podUID="3515275e-be1d-4ebe-b7fb-41178d545638"
Nov 02 18:11:11.390228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.691956    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.390228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.701081    4456 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-pgmzg container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused" start-of-body=
Nov 02 18:11:11.390228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.701120    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": dial tcp 10.134.0.12:8080: connect: connection refused"
Nov 02 18:11:11.390228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.714844    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.390228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.714919    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.390228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.715014    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.390228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.715431    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.390698 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.715524    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.390698 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.715908    4456 prober_manager.go:356] "Failed to trigger a manual run" probe="Readiness"
Nov 02 18:11:11.390698 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.725090    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2" podUID="8064da0c-fc5d-493c-bcdf-4327716d4cc2" containerName="migrator" containerID="cri-o://0675abbd85b4030c862f933f71117ffb0019086a01fbec8168e2bf31f2755a25" gracePeriod=30
Nov 02 18:11:11.390698 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.725274    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": dial tcp 10.134.0.124:8443: connect: connection refused"
Nov 02 18:11:11.390698 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.738520    4456 patch_prober.go:29] interesting pod/metrics-server-7d987f75-ntp98 container/metrics-server namespace/openshift-monitoring: Readiness probe status=failure output="Get \"https://10.134.0.16:10250/livez\": dial tcp 10.134.0.16:10250: connect: connection refused" start-of-body=
Nov 02 18:11:11.390698 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.738559    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-monitoring/metrics-server-7d987f75-ntp98" podUID="18d2ae0a-bf65-4aee-8730-093e00b54a24" containerName="metrics-server" probeResult="failure" output="Get \"https://10.134.0.16:10250/livez\": dial tcp 10.134.0.16:10250: connect: connection refused"
Nov 02 18:11:11.390698 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.739465    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.146:8443/health\": dial tcp 10.134.0.146:8443: connect: connection refused"
Nov 02 18:11:11.398844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.739891    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.46:8443/health\": dial tcp 10.134.0.46:8443: connect: connection refused"
Nov 02 18:11:11.398844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.742721    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" probeResult="failure" output="Get \"https://10.134.0.124:8443/health\": dial tcp 10.134.0.124:8443: connect: connection refused"
Nov 02 18:11:11.398844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.774748    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vhqq9"]
Nov 02 18:11:11.398844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.803635    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy" containerID="cri-o://e079ea8bc432c6b15221e1f0f0d1f1094347bece3a5dc356bdb25b749c2da4d5" gracePeriod=30
Nov 02 18:11:11.398844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.808589    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy" containerID="cri-o://0f61b587f093bc2a10aa5b15cd1853e4adc293fe1098512029b4141d26f10d42" gracePeriod=30
Nov 02 18:11:11.398844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.878457    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3515275e-be1d-4ebe-b7fb-41178d545638-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr\" (UID: \"3515275e-be1d-4ebe-b7fb-41178d545638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"
Nov 02 18:11:11.399136 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.888087    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy" containerID="cri-o://6a3d5cd7a3cf30d356953f12d067b1b473d9827087fa6525e30fb999560de281" gracePeriod=30
Nov 02 18:11:11.399136 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.894624    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy" containerID="cri-o://f5f583c97bede29bd199be0ec2003a50248abd83bffd0425ce6e7cb495a11ca3" gracePeriod=30
Nov 02 18:11:11.399136 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895160    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-dns/dns-default-zmrdt"
Nov 02 18:11:11.399136 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895290    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:11:11.399136 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895360    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 18:11:11.399136 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895386    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"]
Nov 02 18:11:11.399136 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895423    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-v2zdq"]
Nov 02 18:11:11.399136 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895469    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 18:11:11.399136 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895513    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 18:11:11.399136 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895565    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:11:11.399551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895608    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 18:11:11.399551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895654    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 18:11:11.399551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895681    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 18:11:11.399551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895731    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 18:11:11.399551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895774    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kube-system/konnectivity-agent-mxsjq"
Nov 02 18:11:11.399551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895796    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 18:11:11.399551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895857    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"
Nov 02 18:11:11.399551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895899    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 18:11:11.399551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895949    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:11:11.399551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.895995    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 18:11:11.399551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896037    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 18:11:11.399998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896085    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 18:11:11.399998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896129    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"
Nov 02 18:11:11.399998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896168    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4"
Nov 02 18:11:11.399998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896216    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"
Nov 02 18:11:11.399998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896257    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 18:11:11.399998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896391    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 18:11:11.399998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896483    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 18:11:11.399998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896559    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 18:11:11.399998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896603    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 18:11:11.399998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896653    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"
Nov 02 18:11:11.399998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896695    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 18:11:11.400433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896733    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 18:11:11.400433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896829    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"
Nov 02 18:11:11.400433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896874    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 18:11:11.400433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896907    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 18:11:11.400433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.896966    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 18:11:11.400433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:10.983214    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v787t\" (UniqueName: \"kubernetes.io/projected/3515275e-be1d-4ebe-b7fb-41178d545638-kube-api-access-v787t\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr\" (UID: \"3515275e-be1d-4ebe-b7fb-41178d545638\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"
Nov 02 18:11:11.400433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.008862    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"
Nov 02 18:11:11.400433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.009235    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 18:11:11.400433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.015258    4456 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:11:11.400433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.020921    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 18:11:11.400850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.025873    4456 scope.go:122] "RemoveContainer" containerID="d9c0ba7de1c37929a22a23628e19d8a7160ddd083595efe4618b8976b53a7039"
Nov 02 18:11:11.400850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.099717    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-v2zdq"
Nov 02 18:11:11.400850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.104301    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vhqq9"
Nov 02 18:11:11.400850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.116781    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-tchlj"
Nov 02 18:11:11.400850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.149858    4456 plugin_watcher.go:214] "Removing socket path from desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 18:11:11.400850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.159679    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" podUID="758467e6-69e4-4c1f-b344-e1e74497c298" containerName="csi-node-driver-registrar" containerID="cri-o://2f2d0ecef34f90d7be65ae3bfde69eebfd2e056a726c73ec5c4cc23ac48cb253" gracePeriod=30
Nov 02 18:11:11.400850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.223383    4456 kubelet.go:1689] "Image garbage collection succeeded"
Nov 02 18:11:11.400850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.225687    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-driver/0.log"
Nov 02 18:11:11.400850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.344880    4456 generic.go:356] "Generic (PLEG): container finished" podID="8064da0c-fc5d-493c-bcdf-4327716d4cc2" containerID="0675abbd85b4030c862f933f71117ffb0019086a01fbec8168e2bf31f2755a25" exitCode=0
Nov 02 18:11:11.400850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.345005    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2" event={"ID":"8064da0c-fc5d-493c-bcdf-4327716d4cc2","Type":"ContainerDied","Data":"0675abbd85b4030c862f933f71117ffb0019086a01fbec8168e2bf31f2755a25"}
Nov 02 18:11:11.401275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.350746    4456 generic.go:356] "Generic (PLEG): container finished" podID="ec905c34-6dfb-483f-bb3d-7ba412262584" containerID="a3e5b342fbd24d60a087ce776f7958683a916f3ab12b9d0f407b62ce53f0987a" exitCode=0
Nov 02 18:11:11.401275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.350858    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" event={"ID":"ec905c34-6dfb-483f-bb3d-7ba412262584","Type":"ContainerDied","Data":"a3e5b342fbd24d60a087ce776f7958683a916f3ab12b9d0f407b62ce53f0987a"}
Nov 02 18:11:11.401275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.350914    4456 scope.go:122] "RemoveContainer" containerID="98bb1cdd787c025aa16ef0cb488e4b052f1a1c499a91687b7d53b720295e0af2"
Nov 02 18:11:11.401275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.352422    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:11.401275 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.364122    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a29999: configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.401275 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.364190    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume podName:1c190514-ba33-4844-a9ec-e4bb47b5f706 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:15.364169588 +0000 UTC m=+2491.855232752 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume") pod "payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" (UID: "1c190514-ba33-4844-a9ec-e4bb47b5f706") : configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.401275 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.364237    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-872354: configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.401601 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.364271    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume podName:cf85b137-2b1e-4e22-8f6e-744a02ea4e46 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:15.364260896 +0000 UTC m=+2491.855324041 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume") pod "payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" (UID: "cf85b137-2b1e-4e22-8f6e-744a02ea4e46") : configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.401601 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.364308    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a5c1da: configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.401601 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.364338    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume podName:96db7b02-b878-435a-869e-1d30e147c61d nodeName:}" failed. No retries permitted until 2026-11-02 18:11:15.364328865 +0000 UTC m=+2491.855392025 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" (UID: "96db7b02-b878-435a-869e-1d30e147c61d") : configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.401601 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.364376    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-2c9b1d: configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.401780 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.364406    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume podName:3d96e066-10c0-4fe2-9e24-ffd6195b22af nodeName:}" failed. No retries permitted until 2026-11-02 18:11:15.364396683 +0000 UTC m=+2491.855459828 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" (UID: "3d96e066-10c0-4fe2-9e24-ffd6195b22af") : configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.401780 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.364444    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-96768b: configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.401780 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.364474    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume podName:22b556c1-112b-49d4-932c-0143e912a54e nodeName:}" failed. No retries permitted until 2026-11-02 18:11:15.36446495 +0000 UTC m=+2491.855528091 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" (UID: "22b556c1-112b-49d4-932c-0143e912a54e") : configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.401780 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.364511    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-96768b: configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.401976 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.364542    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume podName:5ce3335c-36e6-4018-a264-5b79085f09f3 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:15.364531734 +0000 UTC m=+2491.855594878 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume") pod "payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" (UID: "5ce3335c-36e6-4018-a264-5b79085f09f3") : configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:11.401976 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.364576    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a29999: configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.401976 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.364605    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume podName:7e09749e-e648-40a3-a5f1-f0dc35fccf24 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:15.364595666 +0000 UTC m=+2491.855658812 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" (UID: "7e09749e-e648-40a3-a5f1-f0dc35fccf24") : configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:11.401976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.370072    4456 generic.go:356] "Generic (PLEG): container finished" podID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerID="b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436" exitCode=0
Nov 02 18:11:11.402165 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.370245    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" event={"ID":"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f","Type":"ContainerDied","Data":"b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436"}
Nov 02 18:11:11.402165 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.376478    4456 generic.go:356] "Generic (PLEG): container finished" podID="a1ea3dac-7073-4564-b247-89ea94a64342" containerID="f46a45f18a610493bf46d8787ee563938a877ab2e179a2e516f44f6e70fcc573" exitCode=0
Nov 02 18:11:11.402165 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.376565    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" event={"ID":"a1ea3dac-7073-4564-b247-89ea94a64342","Type":"ContainerDied","Data":"f46a45f18a610493bf46d8787ee563938a877ab2e179a2e516f44f6e70fcc573"}
Nov 02 18:11:11.485160 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.484993    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-872354: configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.485160 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.485096    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume podName:44b66f41-99ae-446b-a50a-2ec0b71563a4 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:15.485072721 +0000 UTC m=+2491.976135874 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" (UID: "44b66f41-99ae-446b-a50a-2ec0b71563a4") : configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:11.488090 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.485608    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a5c1da: configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.488090 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.485696    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume podName:8c60dfaa-dabe-4726-91b5-165443ff3971 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:15.485679231 +0000 UTC m=+2491.976742395 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume") pod "payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" (UID: "8c60dfaa-dabe-4726-91b5-165443ff3971") : configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:11.489006 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.488973    4456 generic.go:356] "Generic (PLEG): container finished" podID="6957a218-62fc-4dd8-892e-4477b08c046f" containerID="50594c76d72639934489d5c1e00e5b464a34d6a50aa2d64165ad25bf4c978a07" exitCode=1
Nov 02 18:11:11.489302 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.489277    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" event={"ID":"6957a218-62fc-4dd8-892e-4477b08c046f","Type":"ContainerDied","Data":"50594c76d72639934489d5c1e00e5b464a34d6a50aa2d64165ad25bf4c978a07"}
Nov 02 18:11:11.500873 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.500852    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-driver/0.log"
Nov 02 18:11:11.500964 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.500886    4456 generic.go:356] "Generic (PLEG): container finished" podID="758467e6-69e4-4c1f-b344-e1e74497c298" containerID="ad864ff6740e4c778f72c8079e4f03eacc7840d208ff11512bdc1dc1655e8c7a" exitCode=2
Nov 02 18:11:11.501010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.500966    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" event={"ID":"758467e6-69e4-4c1f-b344-e1e74497c298","Type":"ContainerDied","Data":"ad864ff6740e4c778f72c8079e4f03eacc7840d208ff11512bdc1dc1655e8c7a"}
Nov 02 18:11:11.507142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.507112    4456 generic.go:356] "Generic (PLEG): container finished" podID="fb44e7b7-e0cb-4270-b14c-9c2397f37269" containerID="61f0d172af153366e5fe0177879402cf8b046838c4d8800dd144c8d0095dd565" exitCode=0
Nov 02 18:11:11.507246 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.507222    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" event={"ID":"fb44e7b7-e0cb-4270-b14c-9c2397f37269","Type":"ContainerDied","Data":"61f0d172af153366e5fe0177879402cf8b046838c4d8800dd144c8d0095dd565"}
Nov 02 18:11:11.515725 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.515686    4456 generic.go:356] "Generic (PLEG): container finished" podID="cae854c8-88f7-4201-b5d7-4e7c59fdf833" containerID="5b3b2a55035fb65589abdc8d0eb062b82d79bbb8b490a8438c8c6505f3ee9c32" exitCode=0
Nov 02 18:11:11.515982 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.515962    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" event={"ID":"cae854c8-88f7-4201-b5d7-4e7c59fdf833","Type":"ContainerDied","Data":"5b3b2a55035fb65589abdc8d0eb062b82d79bbb8b490a8438c8c6505f3ee9c32"}
Nov 02 18:11:11.517114 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.516212    4456 reconciler.go:150] "OperationExecutor.UnregisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"0001-01-01T00:00:00Z","UUID":"47a14e57-d0eb-4a1c-b8e1-d66369b29c2c","Handler":{},"Name":"ebs.csi.aws.com","Endpoint":"/var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock"}
Nov 02 18:11:11.517114 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.516537    4456 csi_plugin.go:271] kubernetes.io/csi: registrationHandler.DeRegisterPlugin request for plugin ebs.csi.aws.com, endpoint /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 18:11:11.524212 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.524183    4456 generic.go:356] "Generic (PLEG): container finished" podID="e73be583-2a26-4379-8169-0a6b8784a11b" containerID="d381788d1f97c945f12e6a0f9ca8aea3a3f0d92249a339bd95400d14c394ce40" exitCode=1
Nov 02 18:11:11.524355 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.524338    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hr8vz" event={"ID":"e73be583-2a26-4379-8169-0a6b8784a11b","Type":"ContainerDied","Data":"d381788d1f97c945f12e6a0f9ca8aea3a3f0d92249a339bd95400d14c394ce40"}
Nov 02 18:11:11.527140 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.525754    4456 scope.go:122] "RemoveContainer" containerID="d381788d1f97c945f12e6a0f9ca8aea3a3f0d92249a339bd95400d14c394ce40"
Nov 02 18:11:11.527140 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.526483    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-controller\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cert-manager-controller pod=cert-manager-597b95cb84-hr8vz_cert-manager(e73be583-2a26-4379-8169-0a6b8784a11b)\"" pod="cert-manager/cert-manager-597b95cb84-hr8vz" podUID="e73be583-2a26-4379-8169-0a6b8784a11b"
Nov 02 18:11:11.528872 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.528848    4456 generic.go:356] "Generic (PLEG): container finished" podID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerID="e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317" exitCode=0
Nov 02 18:11:11.529068 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.529052    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" event={"ID":"6ebd3467-841a-4331-b149-8acf6e60c12c","Type":"ContainerDied","Data":"e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317"}
Nov 02 18:11:11.532499 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.532471    4456 generic.go:356] "Generic (PLEG): container finished" podID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerID="e4828365cff34649b08e97b25ad67554fb1835933f1008be8e43bed2f20e7560" exitCode=0
Nov 02 18:11:11.532727 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.532702    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" event={"ID":"9051e10f-854f-4cc8-90a7-dda319c49c9a","Type":"ContainerDied","Data":"e4828365cff34649b08e97b25ad67554fb1835933f1008be8e43bed2f20e7560"}
Nov 02 18:11:11.533119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.533099    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"
Nov 02 18:11:11.537016 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.536994    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-zh652"
Nov 02 18:11:11.547379 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.547353    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-v2zdq"
Nov 02 18:11:11.549732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.549700    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vhqq9"
Nov 02 18:11:11.555799 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.555005    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-tchlj"
Nov 02 18:11:11.590245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.590218    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-zh652"]
Nov 02 18:11:11.605671 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.605645    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"
Nov 02 18:11:11.631980 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.631863    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-2c9b1d: configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.632208 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.632193    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume podName:60c0e838-5c79-4bf7-a7c6-87903ae8bd8c nodeName:}" failed. No retries permitted until 2026-11-02 18:11:15.632159393 +0000 UTC m=+2492.123222548 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume") pod "payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" (UID: "60c0e838-5c79-4bf7-a7c6-87903ae8bd8c") : configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:11.723879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.723841    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3515275e-be1d-4ebe-b7fb-41178d545638-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3515275e-be1d-4ebe-b7fb-41178d545638-maas-api-openshift-ca\") pod \"3515275e-be1d-4ebe-b7fb-41178d545638\" (UID: \"3515275e-be1d-4ebe-b7fb-41178d545638\") "
Nov 02 18:11:11.726529 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.726499    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3515275e-be1d-4ebe-b7fb-41178d545638-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3515275e-be1d-4ebe-b7fb-41178d545638-maas-api-xks-ca\") pod \"3515275e-be1d-4ebe-b7fb-41178d545638\" (UID: \"3515275e-be1d-4ebe-b7fb-41178d545638\") "
Nov 02 18:11:11.726983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.726959    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3515275e-be1d-4ebe-b7fb-41178d545638-kube-api-access-v787t\" (UniqueName: \"kubernetes.io/projected/3515275e-be1d-4ebe-b7fb-41178d545638-kube-api-access-v787t\") pod \"3515275e-be1d-4ebe-b7fb-41178d545638\" (UID: \"3515275e-be1d-4ebe-b7fb-41178d545638\") "
Nov 02 18:11:11.728785 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.724404    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3515275e-be1d-4ebe-b7fb-41178d545638-maas-api-openshift-ca" pod "3515275e-be1d-4ebe-b7fb-41178d545638" (UID: "3515275e-be1d-4ebe-b7fb-41178d545638"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:11.831653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.831588    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/3515275e-be1d-4ebe-b7fb-41178d545638-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:11.889781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.888643    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6773e5d1-439f-46b2-b48f-eb8cc957566d" path="/var/lib/kubelet/pods/6773e5d1-439f-46b2-b48f-eb8cc957566d/volumes"
Nov 02 18:11:11.889781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.889560    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bb80eb43-a6a5-4388-8436-6618c2f1493d" path="/var/lib/kubelet/pods/bb80eb43-a6a5-4388-8436-6618c2f1493d/volumes"
Nov 02 18:11:11.897103 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.897074    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-tchlj"]
Nov 02 18:11:11.935803 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.935767    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-d5b905: configmap "payload-processing-plugins-e2e-inf-b-d5b905" not found
Nov 02 18:11:11.935993 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.935904    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume podName:e141d845-db9d-4f94-b05e-cfb7848e8fef nodeName:}" failed. No retries permitted until 2026-11-02 18:11:15.935884691 +0000 UTC m=+2492.426947846 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume") pod "payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" (UID: "e141d845-db9d-4f94-b05e-cfb7848e8fef") : configmap "payload-processing-plugins-e2e-inf-b-d5b905" not found
Nov 02 18:11:11.935993 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.935957    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-fbbbf8: configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.935993 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.935991    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume podName:85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:15.935980992 +0000 UTC m=+2492.427044137 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume") pod "payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" (UID: "85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3") : configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.936145 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.936026    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-bb6d2e: configmap "payload-processing-plugins-e2e-inf-a-bb6d2e" not found
Nov 02 18:11:11.936145 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.936056    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume podName:f4ee5c50-5738-496a-b5ef-c86ead76e0f6 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:15.936046714 +0000 UTC m=+2492.427109871 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume") pod "payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" (UID: "f4ee5c50-5738-496a-b5ef-c86ead76e0f6") : configmap "payload-processing-plugins-e2e-inf-a-bb6d2e" not found
Nov 02 18:11:11.936145 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.936092    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-fbbbf8: configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.936145 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:11.936120    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume podName:3d398f17-735a-4d4b-b1cc-5b24fd266535 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:15.936111318 +0000 UTC m=+2492.427174478 (durationBeforeRetry 4s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" (UID: "3d398f17-735a-4d4b-b1cc-5b24fd266535") : configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:11.959837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.959788    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3515275e-be1d-4ebe-b7fb-41178d545638-maas-api-xks-ca" pod "3515275e-be1d-4ebe-b7fb-41178d545638" (UID: "3515275e-be1d-4ebe-b7fb-41178d545638"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:11:11.960006 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:11.959987    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3515275e-be1d-4ebe-b7fb-41178d545638-kube-api-access-v787t" pod "3515275e-be1d-4ebe-b7fb-41178d545638" (UID: "3515275e-be1d-4ebe-b7fb-41178d545638"). InnerVolumeSpecName "kube-api-access-v787t". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:12.054221 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.054160    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/3515275e-be1d-4ebe-b7fb-41178d545638-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:12.054221 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.054194    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v787t\" (UniqueName: \"kubernetes.io/projected/3515275e-be1d-4ebe-b7fb-41178d545638-kube-api-access-v787t\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:12.415373 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.415343    4456 scope.go:122] "RemoveContainer" containerID="3041e8bd23c8eb7470f8dfba16bd766c148c2bcbf3b2edae11533a65a4f804ed"
Nov 02 18:11:12.559945 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.559859    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-pgmzg" event={"ID":"6e78d6c7-c325-48f2-af90-804dade84d8f","Type":"ContainerStarted","Data":"3736c6b20b77a62af51ed11e3d510954fe7f012d5fa027eb244cb94b608dd9cb"}
Nov 02 18:11:12.567568 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.567531    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" event={"ID":"a1ea3dac-7073-4564-b247-89ea94a64342","Type":"ContainerStarted","Data":"4a523413662dfad5f79a825bb0d5c61b2430be82132b00a3deb4475053a2e2c9"}
Nov 02 18:11:12.570977 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.570942    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9" event={"ID":"6957a218-62fc-4dd8-892e-4477b08c046f","Type":"ContainerStarted","Data":"45a3dda5ce4c89b2debc9e2d689f003e862d6fd8f261e1de4439942225d45880"}
Nov 02 18:11:12.572066 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.572041    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 18:11:12.574389 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.574366    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/metrics-server-7d987f75-ntp98" event={"ID":"18d2ae0a-bf65-4aee-8730-093e00b54a24","Type":"ContainerStarted","Data":"8e133a7e39070e7e6dd7f6f4dbac9c39e535d7d61db30d38545b80502baa4633"}
Nov 02 18:11:12.588133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.574895    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 18:11:12.588133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.577777    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-driver/0.log"
Nov 02 18:11:12.588133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.577805    4456 generic.go:356] "Generic (PLEG): container finished" podID="758467e6-69e4-4c1f-b344-e1e74497c298" containerID="2f2d0ecef34f90d7be65ae3bfde69eebfd2e056a726c73ec5c4cc23ac48cb253" exitCode=0
Nov 02 18:11:12.588133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.577923    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" event={"ID":"758467e6-69e4-4c1f-b344-e1e74497c298","Type":"ContainerDied","Data":"2f2d0ecef34f90d7be65ae3bfde69eebfd2e056a726c73ec5c4cc23ac48cb253"}
Nov 02 18:11:12.588133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.582447    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg" event={"ID":"fb44e7b7-e0cb-4270-b14c-9c2397f37269","Type":"ContainerStarted","Data":"41bfa5922c3ca8e7d650df5d002152b6bb6d0f59215fb04f876f6aadca70b432"}
Nov 02 18:11:12.588133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.583642    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 18:11:12.592570 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.592252    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-dockercfg-qpmjh\""
Nov 02 18:11:12.596847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.595478    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj" event={"ID":"cae854c8-88f7-4201-b5d7-4e7c59fdf833","Type":"ContainerStarted","Data":"287353deedbb191b190becaf49b321a1a594909d8286ca6552a91d6656e13de9"}
Nov 02 18:11:12.596847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.595536    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 18:11:12.602493 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.602474    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"
Nov 02 18:11:12.603654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.602883    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-cainjector-5c57bbd4fc-9mjv9" event={"ID":"9fe1660e-3733-4c7f-bc5c-3820d84aa143","Type":"ContainerStarted","Data":"3f0628a4d83bb4c145d57c0029bc451d9b65855d763be8dca629b0af2e6eba30"}
Nov 02 18:11:12.604498 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.604429    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/prometheus-operator-admission-webhook-6489d8bf89-q2mbj"
Nov 02 18:11:12.696099 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.696064    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:12.725898 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.724434    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-monitoring\"/\"metrics-server-audit-profiles\""
Nov 02 18:11:12.725898 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.724685    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-tls\""
Nov 02 18:11:12.725898 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.724776    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-client-certs\""
Nov 02 18:11:12.728017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:12.727911    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-monitoring\"/\"metrics-server-8hdubeblopv5q\""
Nov 02 18:11:13.089159 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.089124    4456 scope.go:122] "RemoveContainer" containerID="cdd773afd598fd21cda432d2c443c479f7282ad8cf4c3c694738ed4f0698ba92"
Nov 02 18:11:13.129888 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.126571    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2"
Nov 02 18:11:13.219499 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.219470    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8064da0c-fc5d-493c-bcdf-4327716d4cc2-kube-api-access-kjmhh\" (UniqueName: \"kubernetes.io/projected/8064da0c-fc5d-493c-bcdf-4327716d4cc2-kube-api-access-kjmhh\") pod \"8064da0c-fc5d-493c-bcdf-4327716d4cc2\" (UID: \"8064da0c-fc5d-493c-bcdf-4327716d4cc2\") "
Nov 02 18:11:13.222323 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.222295    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8064da0c-fc5d-493c-bcdf-4327716d4cc2-kube-api-access-kjmhh" pod "8064da0c-fc5d-493c-bcdf-4327716d4cc2" (UID: "8064da0c-fc5d-493c-bcdf-4327716d4cc2"). InnerVolumeSpecName "kube-api-access-kjmhh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:13.266292 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.265590    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:13.327167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.325713    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-kjmhh\" (UniqueName: \"kubernetes.io/projected/8064da0c-fc5d-493c-bcdf-4327716d4cc2-kube-api-access-kjmhh\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:13.358505 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.358432    4456 scope.go:122] "RemoveContainer" containerID="0cb78ab8ebb38230ce58498878fbee1c9524a897317015a5ff7bf09d770bfdec"
Nov 02 18:11:13.493668 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.493641    4456 scope.go:122] "RemoveContainer" containerID="31ea127575345103ee79ddb9337e7680f26be2c47d1f8b1c1b65da3c280b1211"
Nov 02 18:11:13.535236 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.535158    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:13.651179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.645482    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2"
Nov 02 18:11:13.651179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.645525    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2" event={"ID":"8064da0c-fc5d-493c-bcdf-4327716d4cc2","Type":"ContainerDied","Data":"c12ae599e6c930890621622a0cb873d074ecc075f38e97c1aba91f32c510fdd8"}
Nov 02 18:11:13.651679 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.651558    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 18:11:13.653215 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.652442    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server" containerID="cri-o://3736c6b20b77a62af51ed11e3d510954fe7f012d5fa027eb244cb94b608dd9cb" gracePeriod=5
Nov 02 18:11:13.691071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.691037    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:13.756011 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.755974    4456 kubelet.go:2727] "SyncLoop (probe)" probe="liveness" status="unhealthy" pod="cert-manager/cert-manager-597b95cb84-hr8vz"
Nov 02 18:11:13.758588 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.758562    4456 scope.go:122] "RemoveContainer" containerID="d381788d1f97c945f12e6a0f9ca8aea3a3f0d92249a339bd95400d14c394ce40"
Nov 02 18:11:13.759148 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:13.759118    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cert-manager-controller\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cert-manager-controller pod=cert-manager-597b95cb84-hr8vz_cert-manager(e73be583-2a26-4379-8169-0a6b8784a11b)\"" pod="cert-manager/cert-manager-597b95cb84-hr8vz" podUID="e73be583-2a26-4379-8169-0a6b8784a11b"
Nov 02 18:11:13.771061 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.771039    4456 scope.go:122] "RemoveContainer" containerID="f102af8d8427f21ccd5327dba082268df8744b692aa3dbb2752b19099048e51c"
Nov 02 18:11:13.877810 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.877021    4456 patch_prober.go:29] interesting pod/downloads-6dd789c8b7-pgmzg container/download-server namespace/openshift-console: Readiness probe status=failure output="Get \"http://10.134.0.12:8080/\": EOF" start-of-body=
Nov 02 18:11:13.877810 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:13.877073    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-console/downloads-6dd789c8b7-pgmzg" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server" probeResult="failure" output="Get \"http://10.134.0.12:8080/\": EOF"
Nov 02 18:11:14.197014 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.196967    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"]
Nov 02 18:11:14.204872 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.204842    4456 scope.go:122] "RemoveContainer" containerID="e4e31f901eface112badee8c3415cf9f3881ca46d033fd3f068c3ec2f49f9ab9"
Nov 02 18:11:14.210266 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.210244    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-s4fkr"]
Nov 02 18:11:14.337871 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.337751    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-tchlj"]
Nov 02 18:11:14.346364 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.346334    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-tchlj"]
Nov 02 18:11:14.365493 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.365407    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:14.375213 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.375188    4456 scope.go:122] "RemoveContainer" containerID="d337c60ba714489232e4f5ede5270e793cd4779290b761e8f42568d05e170851"
Nov 02 18:11:14.403707 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.403568    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-zh652"]
Nov 02 18:11:14.417224 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.416680    4456 scope.go:122] "RemoveContainer" containerID="0675abbd85b4030c862f933f71117ffb0019086a01fbec8168e2bf31f2755a25"
Nov 02 18:11:14.417224 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.417171    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-zh652"]
Nov 02 18:11:14.519008 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.518961    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vhqq9"]
Nov 02 18:11:14.533183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.533154    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vhqq9"]
Nov 02 18:11:14.675729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.675047    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" event={"ID":"6ebd3467-841a-4331-b149-8acf6e60c12c","Type":"ContainerStarted","Data":"0acda9a9044766402073a7c3d99977dbfc769a9c9271311f9a6c2b863409852a"}
Nov 02 18:11:14.676740 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.676640    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 18:11:14.700883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.699747    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-v2zdq"]
Nov 02 18:11:14.722580 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.722543    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-v2zdq"]
Nov 02 18:11:14.735576 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.735540    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" event={"ID":"9051e10f-854f-4cc8-90a7-dda319c49c9a","Type":"ContainerStarted","Data":"39ff5c46888272351315f2d7d8139cbf2a129442737970d9469e7b5657c07974"}
Nov 02 18:11:14.738708 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.738678    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 18:11:14.744939 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.744909    4456 generic.go:356] "Generic (PLEG): container finished" podID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerID="3736c6b20b77a62af51ed11e3d510954fe7f012d5fa027eb244cb94b608dd9cb" exitCode=0
Nov 02 18:11:14.745061 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.745040    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-pgmzg" event={"ID":"6e78d6c7-c325-48f2-af90-804dade84d8f","Type":"ContainerDied","Data":"3736c6b20b77a62af51ed11e3d510954fe7f012d5fa027eb244cb94b608dd9cb"}
Nov 02 18:11:14.745124 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.745102    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-console/downloads-6dd789c8b7-pgmzg" event={"ID":"6e78d6c7-c325-48f2-af90-804dade84d8f","Type":"ContainerDied","Data":"0fc277289305105311582e26f2bdf721ba404bf132d41beaef4f4aa7518b9ace"}
Nov 02 18:11:14.745170 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.745129    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="0fc277289305105311582e26f2bdf721ba404bf132d41beaef4f4aa7518b9ace"
Nov 02 18:11:14.745170 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.745156    4456 scope.go:122] "RemoveContainer" containerID="4cee89e694b5799548d6085cbcaa2612948ac905b236f27ac06a462827cad81c"
Nov 02 18:11:14.754182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.754146    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462" event={"ID":"ec905c34-6dfb-483f-bb3d-7ba412262584","Type":"ContainerStarted","Data":"2068d48c00792b758e1ed737fc3768d1e3dfc21ca5052f95a22c69bbfc3dd22b"}
Nov 02 18:11:14.755586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.755565    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 18:11:14.761916 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.759561    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" event={"ID":"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f","Type":"ContainerStarted","Data":"b1443853392ccb3e8ec99e7b4b965424d6ddec7a68fbdb6bf5bf89d1a1f57073"}
Nov 02 18:11:14.761916 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.759988    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 18:11:14.767345 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.767319    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-pgmzg"
Nov 02 18:11:14.822985 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.822958    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6e78d6c7-c325-48f2-af90-804dade84d8f-tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e78d6c7-c325-48f2-af90-804dade84d8f-tmp\") pod \"6e78d6c7-c325-48f2-af90-804dade84d8f\" (UID: \"6e78d6c7-c325-48f2-af90-804dade84d8f\") "
Nov 02 18:11:14.823413 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.823362    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6e78d6c7-c325-48f2-af90-804dade84d8f-kube-api-access-7s4l4\" (UniqueName: \"kubernetes.io/projected/6e78d6c7-c325-48f2-af90-804dade84d8f-kube-api-access-7s4l4\") pod \"6e78d6c7-c325-48f2-af90-804dade84d8f\" (UID: \"6e78d6c7-c325-48f2-af90-804dade84d8f\") "
Nov 02 18:11:14.828199 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.828176    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6e78d6c7-c325-48f2-af90-804dade84d8f-tmp" pod "6e78d6c7-c325-48f2-af90-804dade84d8f" (UID: "6e78d6c7-c325-48f2-af90-804dade84d8f"). InnerVolumeSpecName "tmp". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:14.828424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.828404    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6e78d6c7-c325-48f2-af90-804dade84d8f-kube-api-access-7s4l4" pod "6e78d6c7-c325-48f2-af90-804dade84d8f" (UID: "6e78d6c7-c325-48f2-af90-804dade84d8f"). InnerVolumeSpecName "kube-api-access-7s4l4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:14.927449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.927421    4456 reconciler_common.go:299] "Volume detached for volume \"tmp\" (UniqueName: \"kubernetes.io/empty-dir/6e78d6c7-c325-48f2-af90-804dade84d8f-tmp\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:14.927449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.927447    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7s4l4\" (UniqueName: \"kubernetes.io/projected/6e78d6c7-c325-48f2-af90-804dade84d8f-kube-api-access-7s4l4\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:14.984644 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.984552    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2"]
Nov 02 18:11:14.990296 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:14.990257    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-kube-storage-version-migrator/migrator-66df5d496b-lxxf2"]
Nov 02 18:11:15.082647 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.082611    4456 plugin_watcher.go:201] "Adding socket path or updating timestamp to desired state cache" path="/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock"
Nov 02 18:11:15.089011 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.088975    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:15.465545 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.465517    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-96768b: configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:15.465702 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.465582    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume podName:22b556c1-112b-49d4-932c-0143e912a54e nodeName:}" failed. No retries permitted until 2026-11-02 18:11:23.465567146 +0000 UTC m=+2499.956630302 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" (UID: "22b556c1-112b-49d4-932c-0143e912a54e") : configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:15.465789 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.465767    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-96768b: configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:15.465996 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.465936    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a29999: configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:15.465996 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.465988    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume podName:5ce3335c-36e6-4018-a264-5b79085f09f3 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:23.465966326 +0000 UTC m=+2499.957029482 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume") pod "payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" (UID: "5ce3335c-36e6-4018-a264-5b79085f09f3") : configmap "payload-processing-plugins-e2e-inf-a-96768b" not found
Nov 02 18:11:15.466159 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.465999    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a29999: configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:15.466159 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.466023    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume podName:7e09749e-e648-40a3-a5f1-f0dc35fccf24 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:23.466013553 +0000 UTC m=+2499.957076699 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" (UID: "7e09749e-e648-40a3-a5f1-f0dc35fccf24") : configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:15.466159 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.466041    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume podName:1c190514-ba33-4844-a9ec-e4bb47b5f706 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:23.466031443 +0000 UTC m=+2499.957094583 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume") pod "payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" (UID: "1c190514-ba33-4844-a9ec-e4bb47b5f706") : configmap "payload-processing-plugins-e2e-inf-a-a29999" not found
Nov 02 18:11:15.466342 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.466324    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-872354: configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:15.466395 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.466354    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-2c9b1d: configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:15.466395 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.466376    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume podName:cf85b137-2b1e-4e22-8f6e-744a02ea4e46 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:23.466361241 +0000 UTC m=+2499.957424399 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume") pod "payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" (UID: "cf85b137-2b1e-4e22-8f6e-744a02ea4e46") : configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:15.466462 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.466395    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume podName:3d96e066-10c0-4fe2-9e24-ffd6195b22af nodeName:}" failed. No retries permitted until 2026-11-02 18:11:23.466386087 +0000 UTC m=+2499.957449228 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" (UID: "3d96e066-10c0-4fe2-9e24-ffd6195b22af") : configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:15.466462 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.466399    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a5c1da: configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:15.466462 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.466424    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume podName:96db7b02-b878-435a-869e-1d30e147c61d nodeName:}" failed. No retries permitted until 2026-11-02 18:11:23.466416586 +0000 UTC m=+2499.957479725 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" (UID: "96db7b02-b878-435a-869e-1d30e147c61d") : configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:15.490081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.488981    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 18:11:15.521807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.521740    4456 reconciler.go:167] "OperationExecutor.RegisterPlugin started" plugin={"SocketPath":"/var/lib/kubelet/plugins_registry/ebs.csi.aws.com-reg.sock","Timestamp":"2026-11-02T18:11:15.082634088Z","UUID":"782599c6-139a-42c4-9a11-d89b4528019c","Handler":null,"Name":"","Endpoint":""}
Nov 02 18:11:15.522860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.522825    4456 csi_plugin.go:106] kubernetes.io/csi: Trying to validate a new CSI Driver with name: ebs.csi.aws.com endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock versions: 1.0.0
Nov 02 18:11:15.522860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.522860    4456 csi_plugin.go:119] kubernetes.io/csi: Register new plugin with name: ebs.csi.aws.com at endpoint: /var/lib/kubelet/plugins/ebs.csi.aws.com/csi.sock
Nov 02 18:11:15.571220 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.571182    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-872354: configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:15.571354 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.571264    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume podName:44b66f41-99ae-446b-a50a-2ec0b71563a4 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:23.571244564 +0000 UTC m=+2500.062307720 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume") pod "payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" (UID: "44b66f41-99ae-446b-a50a-2ec0b71563a4") : configmap "payload-processing-plugins-e2e-inf-a-872354" not found
Nov 02 18:11:15.573937 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.573909    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-a5c1da: configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:15.574037 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.573974    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume podName:8c60dfaa-dabe-4726-91b5-165443ff3971 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:23.573957361 +0000 UTC m=+2500.065020517 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume") pod "payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" (UID: "8c60dfaa-dabe-4726-91b5-165443ff3971") : configmap "payload-processing-plugins-e2e-inf-a-a5c1da" not found
Nov 02 18:11:15.628184 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.628149    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3515275e-be1d-4ebe-b7fb-41178d545638" path="/var/lib/kubelet/pods/3515275e-be1d-4ebe-b7fb-41178d545638/volumes"
Nov 02 18:11:15.629059 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.629022    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8064da0c-fc5d-493c-bcdf-4327716d4cc2" path="/var/lib/kubelet/pods/8064da0c-fc5d-493c-bcdf-4327716d4cc2/volumes"
Nov 02 18:11:15.680904 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.680866    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-2c9b1d: configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:15.681072 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:15.680966    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume podName:60c0e838-5c79-4bf7-a7c6-87903ae8bd8c nodeName:}" failed. No retries permitted until 2026-11-02 18:11:23.680943706 +0000 UTC m=+2500.172006855 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume") pod "payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" (UID: "60c0e838-5c79-4bf7-a7c6-87903ae8bd8c") : configmap "payload-processing-plugins-e2e-inf-b-2c9b1d" not found
Nov 02 18:11:15.685863 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.685804    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" podUID="44b66f41-99ae-446b-a50a-2ec0b71563a4" containerName="payload-pre-processing" containerID="cri-o://a15fc9b95bc579e3250f894c650e5c31574a16f7a9e567db93f2291e2e8e8b0a" gracePeriod=25
Nov 02 18:11:15.685978 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.685910    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerName="payload-processing" containerID="cri-o://0dcde13a6a52ff4eff6e292ac044743343cbb0afa2e3f9446f8189685a9bd665" gracePeriod=25
Nov 02 18:11:15.686054 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.686027    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" podUID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" containerName="payload-processing" containerID="cri-o://1f50936a9578f37077e2da181b984035ceacd77c575058ecec4a61436289a702" gracePeriod=25
Nov 02 18:11:15.686103 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.685804    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerName="payload-processing" containerID="cri-o://52131bd7b83ca9822e72e558a3dff218d4a6e09dcbe0ddc00be3cd9e069c1cbb" gracePeriod=25
Nov 02 18:11:15.686198 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.686174    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerName="payload-processing" containerID="cri-o://83a5c5da51e2a33c43fedd46148933222917fbcba3a4cacd320bc29364bbf103" gracePeriod=25
Nov 02 18:11:15.686247 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.686217    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerName="payload-pre-processing" containerID="cri-o://6c24194466a92255caca64b6c1b7d27bee17e48cdd5018b681fa9f4ecb1b733e" gracePeriod=25
Nov 02 18:11:15.686301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.686162    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" podUID="22b556c1-112b-49d4-932c-0143e912a54e" containerName="payload-pre-processing" containerID="cri-o://968e2de3ae2524b2141ebfc9d2cc91b4232330bb2b1e8308c1e4e1329d68b932" gracePeriod=25
Nov 02 18:11:15.688723 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.688668    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" podUID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" containerName="payload-pre-processing" containerID="cri-o://4fa143115bc295dd3e06ce938045b94490edee81d1b9aa59b048d0f8a259384b" gracePeriod=25
Nov 02 18:11:15.688954 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.688692    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" podUID="e141d845-db9d-4f94-b05e-cfb7848e8fef" containerName="payload-processing" containerID="cri-o://723c09d5d9658eb778a71667f74238ad51cacca740ec11432caf9b15357688e9" gracePeriod=25
Nov 02 18:11:15.689254 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.688716    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" podUID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" containerName="payload-processing" containerID="cri-o://5eca9c03a08e76e5e6f070cb8052b78663a3818d87186a38c4402f7bf5034995" gracePeriod=25
Nov 02 18:11:15.725028 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.716437    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing" containerID="cri-o://1780545d23b0db3b73d9b9c93d4463de2ecb228acbbac446492792422b835d1c" gracePeriod=25
Nov 02 18:11:15.725028 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.716621    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing" containerID="cri-o://47ce80eead8cafb5ad533cbe8d61db652959180c5d1b75fef0c82fd03c99ce2f" gracePeriod=25
Nov 02 18:11:15.788678 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.788655    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-console/downloads-6dd789c8b7-pgmzg"
Nov 02 18:11:15.792781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.792760    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-driver/0.log"
Nov 02 18:11:15.792982 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.792956    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" event={"ID":"758467e6-69e4-4c1f-b344-e1e74497c298","Type":"ContainerStarted","Data":"b7982b02f806d7849a26dea268a81e3d6fc4b1be48d9167f1b380cb60070db6a"}
Nov 02 18:11:15.793053 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.793041    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-cluster-csi-drivers/aws-ebs-csi-driver-node-6bb2n" event={"ID":"758467e6-69e4-4c1f-b344-e1e74497c298","Type":"ContainerStarted","Data":"d672c2960557e0fac7a9cbd4ebdfca62ad03bd92b49cd123b6a661c05b3fc514"}
Nov 02 18:11:15.855620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.855540    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerName="payload-pre-processing" containerID="cri-o://8c6acffecf9ac35beb62fec780633f1baf430b68dfeefab50a43468a043033a3" gracePeriod=25
Nov 02 18:11:15.996900 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:15.996122    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerName="payload-processing" containerID="cri-o://e8fac6b06137604d986996a69eb0437546b5e8e988303a1180286636239ba239" gracePeriod=25
Nov 02 18:11:16.080637 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:16.055529    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-fbbbf8: configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:16.080637 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:16.055608    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume podName:85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:24.055589657 +0000 UTC m=+2500.546652816 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume") pod "payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" (UID: "85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3") : configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:16.080637 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:16.058630    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-fbbbf8: configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:16.080637 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:16.058721    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume podName:3d398f17-735a-4d4b-b1cc-5b24fd266535 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:24.058690992 +0000 UTC m=+2500.549754138 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume") pod "payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" (UID: "3d398f17-735a-4d4b-b1cc-5b24fd266535") : configmap "payload-processing-plugins-e2e-inf-b-fbbbf8" not found
Nov 02 18:11:16.080637 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:16.058775    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-a-bb6d2e: configmap "payload-processing-plugins-e2e-inf-a-bb6d2e" not found
Nov 02 18:11:16.096103 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:16.058833    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume podName:f4ee5c50-5738-496a-b5ef-c86ead76e0f6 nodeName:}" failed. No retries permitted until 2026-11-02 18:11:24.058800832 +0000 UTC m=+2500.549863986 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume") pod "payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" (UID: "f4ee5c50-5738-496a-b5ef-c86ead76e0f6") : configmap "payload-processing-plugins-e2e-inf-a-bb6d2e" not found
Nov 02 18:11:16.096103 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:16.058875    4456 configmap.go:193] Couldn't get configMap openshift-ingress/payload-processing-plugins-e2e-inf-b-d5b905: configmap "payload-processing-plugins-e2e-inf-b-d5b905" not found
Nov 02 18:11:16.096103 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:16.058903    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume podName:e141d845-db9d-4f94-b05e-cfb7848e8fef nodeName:}" failed. No retries permitted until 2026-11-02 18:11:24.0588946 +0000 UTC m=+2500.549957741 (durationBeforeRetry 8s). Error: MountVolume.SetUp failed for volume "plugins-config-volume" (UniqueName: "kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume") pod "payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" (UID: "e141d845-db9d-4f94-b05e-cfb7848e8fef") : configmap "payload-processing-plugins-e2e-inf-b-d5b905" not found
Nov 02 18:11:16.199207 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.197666    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:16.403907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.377461    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:16.560885 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.552333    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:16.648969 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:16.639263    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:11:16.706051 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.705969    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:16.823509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.823480    4456 generic.go:356] "Generic (PLEG): container finished" podID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerID="47ce80eead8cafb5ad533cbe8d61db652959180c5d1b75fef0c82fd03c99ce2f" exitCode=0
Nov 02 18:11:16.823729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.823708    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" event={"ID":"8c60dfaa-dabe-4726-91b5-165443ff3971","Type":"ContainerDied","Data":"47ce80eead8cafb5ad533cbe8d61db652959180c5d1b75fef0c82fd03c99ce2f"}
Nov 02 18:11:16.828123 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.828100    4456 generic.go:356] "Generic (PLEG): container finished" podID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerID="52131bd7b83ca9822e72e558a3dff218d4a6e09dcbe0ddc00be3cd9e069c1cbb" exitCode=0
Nov 02 18:11:16.828334 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.828318    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" event={"ID":"f4ee5c50-5738-496a-b5ef-c86ead76e0f6","Type":"ContainerDied","Data":"52131bd7b83ca9822e72e558a3dff218d4a6e09dcbe0ddc00be3cd9e069c1cbb"}
Nov 02 18:11:16.830903 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.830882    4456 generic.go:356] "Generic (PLEG): container finished" podID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerID="8c6acffecf9ac35beb62fec780633f1baf430b68dfeefab50a43468a043033a3" exitCode=0
Nov 02 18:11:16.831910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.831891    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" event={"ID":"7e09749e-e648-40a3-a5f1-f0dc35fccf24","Type":"ContainerDied","Data":"8c6acffecf9ac35beb62fec780633f1baf430b68dfeefab50a43468a043033a3"}
Nov 02 18:11:16.834418 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.834398    4456 generic.go:356] "Generic (PLEG): container finished" podID="e141d845-db9d-4f94-b05e-cfb7848e8fef" containerID="723c09d5d9658eb778a71667f74238ad51cacca740ec11432caf9b15357688e9" exitCode=0
Nov 02 18:11:16.834585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.834566    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" event={"ID":"e141d845-db9d-4f94-b05e-cfb7848e8fef","Type":"ContainerDied","Data":"723c09d5d9658eb778a71667f74238ad51cacca740ec11432caf9b15357688e9"}
Nov 02 18:11:16.840091 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.840053    4456 generic.go:356] "Generic (PLEG): container finished" podID="22b556c1-112b-49d4-932c-0143e912a54e" containerID="968e2de3ae2524b2141ebfc9d2cc91b4232330bb2b1e8308c1e4e1329d68b932" exitCode=0
Nov 02 18:11:16.840342 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.840323    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" event={"ID":"22b556c1-112b-49d4-932c-0143e912a54e","Type":"ContainerDied","Data":"968e2de3ae2524b2141ebfc9d2cc91b4232330bb2b1e8308c1e4e1329d68b932"}
Nov 02 18:11:16.842709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.842689    4456 generic.go:356] "Generic (PLEG): container finished" podID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerID="6c24194466a92255caca64b6c1b7d27bee17e48cdd5018b681fa9f4ecb1b733e" exitCode=0
Nov 02 18:11:16.842946 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.842927    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" event={"ID":"3d398f17-735a-4d4b-b1cc-5b24fd266535","Type":"ContainerDied","Data":"6c24194466a92255caca64b6c1b7d27bee17e48cdd5018b681fa9f4ecb1b733e"}
Nov 02 18:11:16.845703 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.845484    4456 generic.go:356] "Generic (PLEG): container finished" podID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" containerID="5eca9c03a08e76e5e6f070cb8052b78663a3818d87186a38c4402f7bf5034995" exitCode=0
Nov 02 18:11:16.845703 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.845557    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" event={"ID":"60c0e838-5c79-4bf7-a7c6-87903ae8bd8c","Type":"ContainerDied","Data":"5eca9c03a08e76e5e6f070cb8052b78663a3818d87186a38c4402f7bf5034995"}
Nov 02 18:11:16.887846 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.874620    4456 generic.go:356] "Generic (PLEG): container finished" podID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerID="83a5c5da51e2a33c43fedd46148933222917fbcba3a4cacd320bc29364bbf103" exitCode=0
Nov 02 18:11:16.887846 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.874719    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" event={"ID":"85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3","Type":"ContainerDied","Data":"83a5c5da51e2a33c43fedd46148933222917fbcba3a4cacd320bc29364bbf103"}
Nov 02 18:11:16.930509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.929525    4456 generic.go:356] "Generic (PLEG): container finished" podID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerID="7b5ffd17510b581a401cae0b3017c71076c1341b3ae93eed0e79ed9040740ec5" exitCode=0
Nov 02 18:11:16.930509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.929865    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" event={"ID":"3c8d9150-66d1-4be8-8bca-3782fb0eac36","Type":"ContainerDied","Data":"7b5ffd17510b581a401cae0b3017c71076c1341b3ae93eed0e79ed9040740ec5"}
Nov 02 18:11:16.960578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.960543    4456 generic.go:356] "Generic (PLEG): container finished" podID="32802945-5e87-4339-a7eb-33212aab7ba5" containerID="cae3f46a8bc87e7369f5ef014d19a5821ecaa21b04f97dd5ed71d6ee6eb99acf" exitCode=0
Nov 02 18:11:16.961027 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.960643    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" event={"ID":"32802945-5e87-4339-a7eb-33212aab7ba5","Type":"ContainerDied","Data":"cae3f46a8bc87e7369f5ef014d19a5821ecaa21b04f97dd5ed71d6ee6eb99acf"}
Nov 02 18:11:16.961027 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.960884    4456 scope.go:122] "RemoveContainer" containerID="be2b1174a81ac8ec0f45fb11bddb3dc43efc0e67214ba6610f1874b6ec7eb231"
Nov 02 18:11:16.969301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.969218    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" event={"ID":"5ce3335c-36e6-4018-a264-5b79085f09f3","Type":"ContainerDied","Data":"0dcde13a6a52ff4eff6e292ac044743343cbb0afa2e3f9446f8189685a9bd665"}
Nov 02 18:11:16.969437 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.969182    4456 generic.go:356] "Generic (PLEG): container finished" podID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerID="0dcde13a6a52ff4eff6e292ac044743343cbb0afa2e3f9446f8189685a9bd665" exitCode=0
Nov 02 18:11:16.971296 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.971183    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f" event={"ID":"5ce3335c-36e6-4018-a264-5b79085f09f3","Type":"ContainerDied","Data":"928520dc82369ea603f9a3673bc51eaa21c7120c28287ee57759d659ede44319"}
Nov 02 18:11:16.971296 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.971262    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="928520dc82369ea603f9a3673bc51eaa21c7120c28287ee57759d659ede44319"
Nov 02 18:11:16.976212 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.976189    4456 generic.go:356] "Generic (PLEG): container finished" podID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerID="e0bbfd6f9c197b1322c631c19d97f927c7c4b56631d9cd54e8dd5474d3b87b5d" exitCode=0
Nov 02 18:11:16.976483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:16.976453    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" event={"ID":"82ba53b4-e91a-4299-9498-f5abcd34576b","Type":"ContainerDied","Data":"e0bbfd6f9c197b1322c631c19d97f927c7c4b56631d9cd54e8dd5474d3b87b5d"}
Nov 02 18:11:17.061857 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.061785    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:17.122197 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.122164    4456 generic.go:356] "Generic (PLEG): container finished" podID="44b66f41-99ae-446b-a50a-2ec0b71563a4" containerID="a15fc9b95bc579e3250f894c650e5c31574a16f7a9e567db93f2291e2e8e8b0a" exitCode=0
Nov 02 18:11:17.123305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.123276    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" event={"ID":"44b66f41-99ae-446b-a50a-2ec0b71563a4","Type":"ContainerDied","Data":"a15fc9b95bc579e3250f894c650e5c31574a16f7a9e567db93f2291e2e8e8b0a"}
Nov 02 18:11:17.132465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.126650    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"
Nov 02 18:11:17.132789 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.130374    4456 generic.go:356] "Generic (PLEG): container finished" podID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" containerID="4fa143115bc295dd3e06ce938045b94490edee81d1b9aa59b048d0f8a259384b" exitCode=0
Nov 02 18:11:17.133136 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.130406    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" event={"ID":"3d96e066-10c0-4fe2-9e24-ffd6195b22af","Type":"ContainerDied","Data":"4fa143115bc295dd3e06ce938045b94490edee81d1b9aa59b048d0f8a259384b"}
Nov 02 18:11:17.136536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.136515    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"
Nov 02 18:11:17.138337 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.138300    4456 generic.go:356] "Generic (PLEG): container finished" podID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" containerID="1f50936a9578f37077e2da181b984035ceacd77c575058ecec4a61436289a702" exitCode=0
Nov 02 18:11:17.138676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.138655    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" event={"ID":"cf85b137-2b1e-4e22-8f6e-744a02ea4e46","Type":"ContainerDied","Data":"1f50936a9578f37077e2da181b984035ceacd77c575058ecec4a61436289a702"}
Nov 02 18:11:17.159759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.159710    4456 generic.go:356] "Generic (PLEG): container finished" podID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerID="f5f583c97bede29bd199be0ec2003a50248abd83bffd0425ce6e7cb495a11ca3" exitCode=0
Nov 02 18:11:17.160420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.160378    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" event={"ID":"67b961eb-ab19-4d8d-bb2a-db7c94a93436","Type":"ContainerDied","Data":"f5f583c97bede29bd199be0ec2003a50248abd83bffd0425ce6e7cb495a11ca3"}
Nov 02 18:11:17.198506 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.189515    4456 generic.go:356] "Generic (PLEG): container finished" podID="96db7b02-b878-435a-869e-1d30e147c61d" containerID="1780545d23b0db3b73d9b9c93d4463de2ecb228acbbac446492792422b835d1c" exitCode=0
Nov 02 18:11:17.200768 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.198870    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" event={"ID":"96db7b02-b878-435a-869e-1d30e147c61d","Type":"ContainerDied","Data":"1780545d23b0db3b73d9b9c93d4463de2ecb228acbbac446492792422b835d1c"}
Nov 02 18:11:17.232903 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.232875    4456 scope.go:122] "RemoveContainer" containerID="0b5a297c1cf1ce7307713189a5bb582c30e4186d062817fc68e564f7bf4411ba"
Nov 02 18:11:17.248863 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.248831    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume\") pod \"85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3\" (UID: \"85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3\") "
Nov 02 18:11:17.249129 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.249108    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5ce3335c-36e6-4018-a264-5b79085f09f3-kube-api-access-ct2fq\" (UniqueName: \"kubernetes.io/projected/5ce3335c-36e6-4018-a264-5b79085f09f3-kube-api-access-ct2fq\") pod \"5ce3335c-36e6-4018-a264-5b79085f09f3\" (UID: \"5ce3335c-36e6-4018-a264-5b79085f09f3\") "
Nov 02 18:11:17.250421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.250401    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-kube-api-access-dv7vb\" (UniqueName: \"kubernetes.io/projected/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-kube-api-access-dv7vb\") pod \"85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3\" (UID: \"85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3\") "
Nov 02 18:11:17.256971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.254189    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume\") pod \"5ce3335c-36e6-4018-a264-5b79085f09f3\" (UID: \"5ce3335c-36e6-4018-a264-5b79085f09f3\") "
Nov 02 18:11:17.256971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.256749    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume" pod "5ce3335c-36e6-4018-a264-5b79085f09f3" (UID: "5ce3335c-36e6-4018-a264-5b79085f09f3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.270798 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.270723    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 18:11:17.275588 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.275304    4456 generic.go:356] "Generic (PLEG): container finished" podID="551602c4-254b-4532-8961-a98bf0f81089" containerID="6a3d5cd7a3cf30d356953f12d067b1b473d9827087fa6525e30fb999560de281" exitCode=0
Nov 02 18:11:17.275588 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.275441    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" event={"ID":"551602c4-254b-4532-8961-a98bf0f81089","Type":"ContainerDied","Data":"6a3d5cd7a3cf30d356953f12d067b1b473d9827087fa6525e30fb999560de281"}
Nov 02 18:11:17.305252 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.304174    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume" pod "85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" (UID: "85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.345022 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.344981    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-kube-api-access-dv7vb" pod "85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" (UID: "85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3"). InnerVolumeSpecName "kube-api-access-dv7vb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.358271 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.358230    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/96db7b02-b878-435a-869e-1d30e147c61d-kube-api-access-nklr4\" (UniqueName: \"kubernetes.io/projected/96db7b02-b878-435a-869e-1d30e147c61d-kube-api-access-nklr4\") pod \"96db7b02-b878-435a-869e-1d30e147c61d\" (UID: \"96db7b02-b878-435a-869e-1d30e147c61d\") "
Nov 02 18:11:17.359401 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.359106    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume\") pod \"96db7b02-b878-435a-869e-1d30e147c61d\" (UID: \"96db7b02-b878-435a-869e-1d30e147c61d\") "
Nov 02 18:11:17.360781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.360747    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/5ce3335c-36e6-4018-a264-5b79085f09f3-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.360948 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.360932    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.361039 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.361028    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dv7vb\" (UniqueName: \"kubernetes.io/projected/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3-kube-api-access-dv7vb\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.362491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.362464    4456 generic.go:356] "Generic (PLEG): container finished" podID="d4161075-440d-4373-b878-57c77612c6d2" containerID="e079ea8bc432c6b15221e1f0f0d1f1094347bece3a5dc356bdb25b749c2da4d5" exitCode=0
Nov 02 18:11:17.369010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.368981    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" event={"ID":"d4161075-440d-4373-b878-57c77612c6d2","Type":"ContainerDied","Data":"e079ea8bc432c6b15221e1f0f0d1f1094347bece3a5dc356bdb25b749c2da4d5"}
Nov 02 18:11:17.385887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.372232    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume" pod "96db7b02-b878-435a-869e-1d30e147c61d" (UID: "96db7b02-b878-435a-869e-1d30e147c61d"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.414307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.395960    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/96db7b02-b878-435a-869e-1d30e147c61d-kube-api-access-nklr4" pod "96db7b02-b878-435a-869e-1d30e147c61d" (UID: "96db7b02-b878-435a-869e-1d30e147c61d"). InnerVolumeSpecName "kube-api-access-nklr4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.414307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.412639    4456 scope.go:122] "RemoveContainer" containerID="17526f5191570a9cb7a3efb0f2691ee4078b3c90dee5e66f8d2131400d0a1ec8"
Nov 02 18:11:17.422365 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.422338    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5ce3335c-36e6-4018-a264-5b79085f09f3-kube-api-access-ct2fq" pod "5ce3335c-36e6-4018-a264-5b79085f09f3" (UID: "5ce3335c-36e6-4018-a264-5b79085f09f3"). InnerVolumeSpecName "kube-api-access-ct2fq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.456259 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.456211    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 18:11:17.462015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.461984    4456 generic.go:356] "Generic (PLEG): container finished" podID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerID="e8fac6b06137604d986996a69eb0437546b5e8e988303a1180286636239ba239" exitCode=0
Nov 02 18:11:17.462198 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.462169    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 18:11:17.462910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.462611    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" event={"ID":"1c190514-ba33-4844-a9ec-e4bb47b5f706","Type":"ContainerDied","Data":"e8fac6b06137604d986996a69eb0437546b5e8e988303a1180286636239ba239"}
Nov 02 18:11:17.467357 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.467319    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 18:11:17.471251 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.471226    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nklr4\" (UniqueName: \"kubernetes.io/projected/96db7b02-b878-435a-869e-1d30e147c61d-kube-api-access-nklr4\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.471401 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.471258    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ct2fq\" (UniqueName: \"kubernetes.io/projected/5ce3335c-36e6-4018-a264-5b79085f09f3-kube-api-access-ct2fq\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.471401 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.471272    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/96db7b02-b878-435a-869e-1d30e147c61d-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.472513 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.472489    4456 generic.go:356] "Generic (PLEG): container finished" podID="0996b2fa-40e8-469c-9ace-75458691fa25" containerID="0f61b587f093bc2a10aa5b15cd1853e4adc293fe1098512029b4141d26f10d42" exitCode=0
Nov 02 18:11:17.473930 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.473902    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" event={"ID":"0996b2fa-40e8-469c-9ace-75458691fa25","Type":"ContainerDied","Data":"0f61b587f093bc2a10aa5b15cd1853e4adc293fe1098512029b4141d26f10d42"}
Nov 02 18:11:17.485618 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.484842    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"
Nov 02 18:11:17.500743 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.497649    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"
Nov 02 18:11:17.504724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.504710    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"
Nov 02 18:11:17.505195 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.505171    4456 scope.go:122] "RemoveContainer" containerID="4a6f1801431eae7eaddf5eebcb23a321305fb236e58f980eafe4d90518d86398"
Nov 02 18:11:17.515854 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.509978    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"
Nov 02 18:11:17.519473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.519408    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"
Nov 02 18:11:17.528387 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.528320    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 18:11:17.535506 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.535481    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 18:11:17.537889 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.537868    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:11:17.543658 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.543633    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 18:11:17.557102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.557076    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 18:11:17.571787 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.571764    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e141d845-db9d-4f94-b05e-cfb7848e8fef-kube-api-access-p9t7z\" (UniqueName: \"kubernetes.io/projected/e141d845-db9d-4f94-b05e-cfb7848e8fef-kube-api-access-p9t7z\") pod \"e141d845-db9d-4f94-b05e-cfb7848e8fef\" (UID: \"e141d845-db9d-4f94-b05e-cfb7848e8fef\") "
Nov 02 18:11:17.572457 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.572436    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume\") pod \"60c0e838-5c79-4bf7-a7c6-87903ae8bd8c\" (UID: \"60c0e838-5c79-4bf7-a7c6-87903ae8bd8c\") "
Nov 02 18:11:17.573040 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.573026    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume\") pod \"f4ee5c50-5738-496a-b5ef-c86ead76e0f6\" (UID: \"f4ee5c50-5738-496a-b5ef-c86ead76e0f6\") "
Nov 02 18:11:17.573497 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.573485    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume\") pod \"3d398f17-735a-4d4b-b1cc-5b24fd266535\" (UID: \"3d398f17-735a-4d4b-b1cc-5b24fd266535\") "
Nov 02 18:11:17.574008 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.573992    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/44b66f41-99ae-446b-a50a-2ec0b71563a4-kube-api-access-qbrdf\" (UniqueName: \"kubernetes.io/projected/44b66f41-99ae-446b-a50a-2ec0b71563a4-kube-api-access-qbrdf\") pod \"44b66f41-99ae-446b-a50a-2ec0b71563a4\" (UID: \"44b66f41-99ae-446b-a50a-2ec0b71563a4\") "
Nov 02 18:11:17.574529 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.574511    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-kube-api-access-f7bnx\" (UniqueName: \"kubernetes.io/projected/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-kube-api-access-f7bnx\") pod \"60c0e838-5c79-4bf7-a7c6-87903ae8bd8c\" (UID: \"60c0e838-5c79-4bf7-a7c6-87903ae8bd8c\") "
Nov 02 18:11:17.574667 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.574654    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-credential-socket\") pod \"82ba53b4-e91a-4299-9498-f5abcd34576b\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") "
Nov 02 18:11:17.574924 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.574912    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-podinfo\") pod \"82ba53b4-e91a-4299-9498-f5abcd34576b\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") "
Nov 02 18:11:17.575253 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.575241    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-workload-certs\") pod \"82ba53b4-e91a-4299-9498-f5abcd34576b\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") "
Nov 02 18:11:17.575343 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.575332    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/22b556c1-112b-49d4-932c-0143e912a54e-kube-api-access-k85vn\" (UniqueName: \"kubernetes.io/projected/22b556c1-112b-49d4-932c-0143e912a54e-kube-api-access-k85vn\") pod \"22b556c1-112b-49d4-932c-0143e912a54e\" (UID: \"22b556c1-112b-49d4-932c-0143e912a54e\") "
Nov 02 18:11:17.575417 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.575408    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-kube-api-access-crv8x\" (UniqueName: \"kubernetes.io/projected/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-kube-api-access-crv8x\") pod \"f4ee5c50-5738-496a-b5ef-c86ead76e0f6\" (UID: \"f4ee5c50-5738-496a-b5ef-c86ead76e0f6\") "
Nov 02 18:11:17.575503 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.575490    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ba53b4-e91a-4299-9498-f5abcd34576b-kube-api-access-gv7wn\" (UniqueName: \"kubernetes.io/projected/82ba53b4-e91a-4299-9498-f5abcd34576b-kube-api-access-gv7wn\") pod \"82ba53b4-e91a-4299-9498-f5abcd34576b\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") "
Nov 02 18:11:17.575706 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.575687    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d96e066-10c0-4fe2-9e24-ffd6195b22af-kube-api-access-rf96w\" (UniqueName: \"kubernetes.io/projected/3d96e066-10c0-4fe2-9e24-ffd6195b22af-kube-api-access-rf96w\") pod \"3d96e066-10c0-4fe2-9e24-ffd6195b22af\" (UID: \"3d96e066-10c0-4fe2-9e24-ffd6195b22af\") "
Nov 02 18:11:17.575967 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.575949    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume\") pod \"44b66f41-99ae-446b-a50a-2ec0b71563a4\" (UID: \"44b66f41-99ae-446b-a50a-2ec0b71563a4\") "
Nov 02 18:11:17.576159 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.576143    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-data\") pod \"82ba53b4-e91a-4299-9498-f5abcd34576b\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") "
Nov 02 18:11:17.576356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.576340    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-token\" (UniqueName: \"kubernetes.io/projected/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-token\") pod \"82ba53b4-e91a-4299-9498-f5abcd34576b\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") "
Nov 02 18:11:17.576469 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.576454    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-envoy\") pod \"82ba53b4-e91a-4299-9498-f5abcd34576b\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") "
Nov 02 18:11:17.576606 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.576590    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3d398f17-735a-4d4b-b1cc-5b24fd266535-kube-api-access-dpl8s\" (UniqueName: \"kubernetes.io/projected/3d398f17-735a-4d4b-b1cc-5b24fd266535-kube-api-access-dpl8s\") pod \"3d398f17-735a-4d4b-b1cc-5b24fd266535\" (UID: \"3d398f17-735a-4d4b-b1cc-5b24fd266535\") "
Nov 02 18:11:17.576786 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.576772    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/82ba53b4-e91a-4299-9498-f5abcd34576b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ba53b4-e91a-4299-9498-f5abcd34576b-istiod-ca-cert\") pod \"82ba53b4-e91a-4299-9498-f5abcd34576b\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") "
Nov 02 18:11:17.576961 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.576944    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-workload-socket\") pod \"82ba53b4-e91a-4299-9498-f5abcd34576b\" (UID: \"82ba53b4-e91a-4299-9498-f5abcd34576b\") "
Nov 02 18:11:17.577073 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.577056    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume\") pod \"3d96e066-10c0-4fe2-9e24-ffd6195b22af\" (UID: \"3d96e066-10c0-4fe2-9e24-ffd6195b22af\") "
Nov 02 18:11:17.577218 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.577202    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume\") pod \"e141d845-db9d-4f94-b05e-cfb7848e8fef\" (UID: \"e141d845-db9d-4f94-b05e-cfb7848e8fef\") "
Nov 02 18:11:17.577370 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.577356    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume\") pod \"22b556c1-112b-49d4-932c-0143e912a54e\" (UID: \"22b556c1-112b-49d4-932c-0143e912a54e\") "
Nov 02 18:11:17.583396 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.575953    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 18:11:17.583800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.572918    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume" pod "60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" (UID: "60c0e838-5c79-4bf7-a7c6-87903ae8bd8c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.583916 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.573433    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume" pod "f4ee5c50-5738-496a-b5ef-c86ead76e0f6" (UID: "f4ee5c50-5738-496a-b5ef-c86ead76e0f6"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.583989 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.573914    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume" pod "3d398f17-735a-4d4b-b1cc-5b24fd266535" (UID: "3d398f17-735a-4d4b-b1cc-5b24fd266535"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.584058 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.575138    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-credential-socket" pod "82ba53b4-e91a-4299-9498-f5abcd34576b" (UID: "82ba53b4-e91a-4299-9498-f5abcd34576b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.584299 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.576253    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e141d845-db9d-4f94-b05e-cfb7848e8fef-kube-api-access-p9t7z" pod "e141d845-db9d-4f94-b05e-cfb7848e8fef" (UID: "e141d845-db9d-4f94-b05e-cfb7848e8fef"). InnerVolumeSpecName "kube-api-access-p9t7z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.584371 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.577860    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-kube-api-access-f7bnx" pod "60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" (UID: "60c0e838-5c79-4bf7-a7c6-87903ae8bd8c"). InnerVolumeSpecName "kube-api-access-f7bnx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.584440 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.578934    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/22b556c1-112b-49d4-932c-0143e912a54e-kube-api-access-k85vn" pod "22b556c1-112b-49d4-932c-0143e912a54e" (UID: "22b556c1-112b-49d4-932c-0143e912a54e"). InnerVolumeSpecName "kube-api-access-k85vn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.584507 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.579005    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/44b66f41-99ae-446b-a50a-2ec0b71563a4-kube-api-access-qbrdf" pod "44b66f41-99ae-446b-a50a-2ec0b71563a4" (UID: "44b66f41-99ae-446b-a50a-2ec0b71563a4"). InnerVolumeSpecName "kube-api-access-qbrdf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.584575 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.579712    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-kube-api-access-crv8x" pod "f4ee5c50-5738-496a-b5ef-c86ead76e0f6" (UID: "f4ee5c50-5738-496a-b5ef-c86ead76e0f6"). InnerVolumeSpecName "kube-api-access-crv8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.584653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.579896    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-workload-certs" pod "82ba53b4-e91a-4299-9498-f5abcd34576b" (UID: "82ba53b4-e91a-4299-9498-f5abcd34576b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.584722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.580403    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-workload-socket" pod "82ba53b4-e91a-4299-9498-f5abcd34576b" (UID: "82ba53b4-e91a-4299-9498-f5abcd34576b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.584791 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.580687    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/82ba53b4-e91a-4299-9498-f5abcd34576b-istiod-ca-cert" pod "82ba53b4-e91a-4299-9498-f5abcd34576b" (UID: "82ba53b4-e91a-4299-9498-f5abcd34576b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.585028 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.581123    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume" pod "44b66f41-99ae-446b-a50a-2ec0b71563a4" (UID: "44b66f41-99ae-446b-a50a-2ec0b71563a4"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.585310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.581175    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume" pod "e141d845-db9d-4f94-b05e-cfb7848e8fef" (UID: "e141d845-db9d-4f94-b05e-cfb7848e8fef"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.585426 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.583079    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume" pod "3d96e066-10c0-4fe2-9e24-ffd6195b22af" (UID: "3d96e066-10c0-4fe2-9e24-ffd6195b22af"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.585530 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.583374    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-data" pod "82ba53b4-e91a-4299-9498-f5abcd34576b" (UID: "82ba53b4-e91a-4299-9498-f5abcd34576b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.585637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.583775    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume" pod "22b556c1-112b-49d4-932c-0143e912a54e" (UID: "22b556c1-112b-49d4-932c-0143e912a54e"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.585720 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.584069    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-token" pod "82ba53b4-e91a-4299-9498-f5abcd34576b" (UID: "82ba53b4-e91a-4299-9498-f5abcd34576b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.585791 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.572906    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 18:11:17.585959 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.585934    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d96e066-10c0-4fe2-9e24-ffd6195b22af-kube-api-access-rf96w" pod "3d96e066-10c0-4fe2-9e24-ffd6195b22af" (UID: "3d96e066-10c0-4fe2-9e24-ffd6195b22af"). InnerVolumeSpecName "kube-api-access-rf96w". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.586630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.586592    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/82ba53b4-e91a-4299-9498-f5abcd34576b-kube-api-access-gv7wn" pod "82ba53b4-e91a-4299-9498-f5abcd34576b" (UID: "82ba53b4-e91a-4299-9498-f5abcd34576b"). InnerVolumeSpecName "kube-api-access-gv7wn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.586716 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.586639    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3d398f17-735a-4d4b-b1cc-5b24fd266535-kube-api-access-dpl8s" pod "3d398f17-735a-4d4b-b1cc-5b24fd266535" (UID: "3d398f17-735a-4d4b-b1cc-5b24fd266535"). InnerVolumeSpecName "kube-api-access-dpl8s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.586927 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.586905    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-podinfo" pod "82ba53b4-e91a-4299-9498-f5abcd34576b" (UID: "82ba53b4-e91a-4299-9498-f5abcd34576b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:17.588007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.587985    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-envoy" pod "82ba53b4-e91a-4299-9498-f5abcd34576b" (UID: "82ba53b4-e91a-4299-9498-f5abcd34576b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.589709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.589693    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 18:11:17.596037 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.594526    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"
Nov 02 18:11:17.599339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.599231    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 18:11:17.683478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.683449    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1c190514-ba33-4844-a9ec-e4bb47b5f706-kube-api-access-b4g6z\" (UniqueName: \"kubernetes.io/projected/1c190514-ba33-4844-a9ec-e4bb47b5f706-kube-api-access-b4g6z\") pod \"1c190514-ba33-4844-a9ec-e4bb47b5f706\" (UID: \"1c190514-ba33-4844-a9ec-e4bb47b5f706\") "
Nov 02 18:11:17.683578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.683507    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-workload-certs\") pod \"551602c4-254b-4532-8961-a98bf0f81089\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") "
Nov 02 18:11:17.683641 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.683580    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-credential-socket\") pod \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") "
Nov 02 18:11:17.683641 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.683629    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-istio-data\") pod \"d4161075-440d-4373-b878-57c77612c6d2\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") "
Nov 02 18:11:17.683727 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.683661    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4161075-440d-4373-b878-57c77612c6d2-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4161075-440d-4373-b878-57c77612c6d2-istiod-ca-cert\") pod \"d4161075-440d-4373-b878-57c77612c6d2\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") "
Nov 02 18:11:17.683727 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.683690    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-istio-data\") pod \"32802945-5e87-4339-a7eb-33212aab7ba5\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") "
Nov 02 18:11:17.683832 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.683756    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-istio-envoy\") pod \"d4161075-440d-4373-b878-57c77612c6d2\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") "
Nov 02 18:11:17.683832 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.683783    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-istio-envoy\") pod \"0996b2fa-40e8-469c-9ace-75458691fa25\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") "
Nov 02 18:11:17.683919 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.683881    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32802945-5e87-4339-a7eb-33212aab7ba5-istio-token\" (UniqueName: \"kubernetes.io/projected/32802945-5e87-4339-a7eb-33212aab7ba5-istio-token\") pod \"32802945-5e87-4339-a7eb-33212aab7ba5\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") "
Nov 02 18:11:17.683919 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.683907    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32802945-5e87-4339-a7eb-33212aab7ba5-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32802945-5e87-4339-a7eb-33212aab7ba5-istiod-ca-cert\") pod \"32802945-5e87-4339-a7eb-33212aab7ba5\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") "
Nov 02 18:11:17.684055 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.683949    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/0996b2fa-40e8-469c-9ace-75458691fa25-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0996b2fa-40e8-469c-9ace-75458691fa25-istio-podinfo\") pod \"0996b2fa-40e8-469c-9ace-75458691fa25\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") "
Nov 02 18:11:17.684055 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.683977    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551602c4-254b-4532-8961-a98bf0f81089-kube-api-access-z2dvq\" (UniqueName: \"kubernetes.io/projected/551602c4-254b-4532-8961-a98bf0f81089-kube-api-access-z2dvq\") pod \"551602c4-254b-4532-8961-a98bf0f81089\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") "
Nov 02 18:11:17.684055 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684040    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-workload-certs\") pod \"0996b2fa-40e8-469c-9ace-75458691fa25\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") "
Nov 02 18:11:17.684219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684068    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/551602c4-254b-4532-8961-a98bf0f81089-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551602c4-254b-4532-8961-a98bf0f81089-istio-podinfo\") pod \"551602c4-254b-4532-8961-a98bf0f81089\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") "
Nov 02 18:11:17.684219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684097    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/d4161075-440d-4373-b878-57c77612c6d2-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4161075-440d-4373-b878-57c77612c6d2-istio-podinfo\") pod \"d4161075-440d-4373-b878-57c77612c6d2\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") "
Nov 02 18:11:17.684219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684121    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-envoy\") pod \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") "
Nov 02 18:11:17.684219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684149    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-workload-certs\") pod \"32802945-5e87-4339-a7eb-33212aab7ba5\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") "
Nov 02 18:11:17.684219 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684171    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-podinfo\") pod \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") "
Nov 02 18:11:17.684478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684192    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-istio-envoy\") pod \"32802945-5e87-4339-a7eb-33212aab7ba5\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") "
Nov 02 18:11:17.684478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684221    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-workload-socket\") pod \"551602c4-254b-4532-8961-a98bf0f81089\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") "
Nov 02 18:11:17.684478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684247    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b961eb-ab19-4d8d-bb2a-db7c94a93436-kube-api-access-6vtvv\" (UniqueName: \"kubernetes.io/projected/67b961eb-ab19-4d8d-bb2a-db7c94a93436-kube-api-access-6vtvv\") pod \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") "
Nov 02 18:11:17.684478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684270    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-istio-data\") pod \"0996b2fa-40e8-469c-9ace-75458691fa25\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") "
Nov 02 18:11:17.684478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684288    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istiod-ca-cert\") pod \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") "
Nov 02 18:11:17.684737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684318    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-kube-api-access-9w876\" (UniqueName: \"kubernetes.io/projected/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-kube-api-access-9w876\") pod \"cf85b137-2b1e-4e22-8f6e-744a02ea4e46\" (UID: \"cf85b137-2b1e-4e22-8f6e-744a02ea4e46\") "
Nov 02 18:11:17.684737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684335    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0996b2fa-40e8-469c-9ace-75458691fa25-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0996b2fa-40e8-469c-9ace-75458691fa25-istiod-ca-cert\") pod \"0996b2fa-40e8-469c-9ace-75458691fa25\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") "
Nov 02 18:11:17.684737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684370    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-credential-socket\") pod \"551602c4-254b-4532-8961-a98bf0f81089\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") "
Nov 02 18:11:17.684737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684393    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-envoy\") pod \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") "
Nov 02 18:11:17.684737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684411    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-workload-certs\") pod \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") "
Nov 02 18:11:17.685106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684442    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-credential-socket\") pod \"d4161075-440d-4373-b878-57c77612c6d2\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") "
Nov 02 18:11:17.685106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684470    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-credential-socket\") pod \"0996b2fa-40e8-469c-9ace-75458691fa25\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") "
Nov 02 18:11:17.685106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684492    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-workload-socket\") pod \"d4161075-440d-4373-b878-57c77612c6d2\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") "
Nov 02 18:11:17.685106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684515    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e09749e-e648-40a3-a5f1-f0dc35fccf24-kube-api-access-f586g\" (UniqueName: \"kubernetes.io/projected/7e09749e-e648-40a3-a5f1-f0dc35fccf24-kube-api-access-f586g\") pod \"7e09749e-e648-40a3-a5f1-f0dc35fccf24\" (UID: \"7e09749e-e648-40a3-a5f1-f0dc35fccf24\") "
Nov 02 18:11:17.685106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684587    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4161075-440d-4373-b878-57c77612c6d2-istio-token\" (UniqueName: \"kubernetes.io/projected/d4161075-440d-4373-b878-57c77612c6d2-istio-token\") pod \"d4161075-440d-4373-b878-57c77612c6d2\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") "
Nov 02 18:11:17.685470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684606    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume\") pod \"8c60dfaa-dabe-4726-91b5-165443ff3971\" (UID: \"8c60dfaa-dabe-4726-91b5-165443ff3971\") "
Nov 02 18:11:17.685470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684635    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4161075-440d-4373-b878-57c77612c6d2-kube-api-access-grksl\" (UniqueName: \"kubernetes.io/projected/d4161075-440d-4373-b878-57c77612c6d2-kube-api-access-grksl\") pod \"d4161075-440d-4373-b878-57c77612c6d2\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") "
Nov 02 18:11:17.685470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684657    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-data\") pod \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") "
Nov 02 18:11:17.685470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684682    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/32802945-5e87-4339-a7eb-33212aab7ba5-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32802945-5e87-4339-a7eb-33212aab7ba5-istio-podinfo\") pod \"32802945-5e87-4339-a7eb-33212aab7ba5\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") "
Nov 02 18:11:17.685470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684699    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-workload-certs\") pod \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") "
Nov 02 18:11:17.685769 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684740    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-workload-socket\") pod \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") "
Nov 02 18:11:17.685769 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684759    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-workload-certs\") pod \"d4161075-440d-4373-b878-57c77612c6d2\" (UID: \"d4161075-440d-4373-b878-57c77612c6d2\") "
Nov 02 18:11:17.685769 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684790    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/551602c4-254b-4532-8961-a98bf0f81089-istio-token\" (UniqueName: \"kubernetes.io/projected/551602c4-254b-4532-8961-a98bf0f81089-istio-token\") pod \"551602c4-254b-4532-8961-a98bf0f81089\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") "
Nov 02 18:11:17.685769 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684839    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-credential-socket\") pod \"32802945-5e87-4339-a7eb-33212aab7ba5\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") "
Nov 02 18:11:17.685769 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684858    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-workload-socket\") pod \"32802945-5e87-4339-a7eb-33212aab7ba5\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") "
Nov 02 18:11:17.686095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684875    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-workload-socket\") pod \"0996b2fa-40e8-469c-9ace-75458691fa25\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") "
Nov 02 18:11:17.686095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684892    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0996b2fa-40e8-469c-9ace-75458691fa25-istio-token\" (UniqueName: \"kubernetes.io/projected/0996b2fa-40e8-469c-9ace-75458691fa25-istio-token\") pod \"0996b2fa-40e8-469c-9ace-75458691fa25\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") "
Nov 02 18:11:17.686095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685039    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume\") pod \"1c190514-ba33-4844-a9ec-e4bb47b5f706\" (UID: \"1c190514-ba33-4844-a9ec-e4bb47b5f706\") "
Nov 02 18:11:17.686095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685110    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8d9150-66d1-4be8-8bca-3782fb0eac36-kube-api-access-2vktq\" (UniqueName: \"kubernetes.io/projected/3c8d9150-66d1-4be8-8bca-3782fb0eac36-kube-api-access-2vktq\") pod \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") "
Nov 02 18:11:17.686095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685168    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/551602c4-254b-4532-8961-a98bf0f81089-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551602c4-254b-4532-8961-a98bf0f81089-istiod-ca-cert\") pod \"551602c4-254b-4532-8961-a98bf0f81089\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") "
Nov 02 18:11:17.686303 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685206    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-podinfo\") pod \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") "
Nov 02 18:11:17.686303 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685275    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-token\" (UniqueName: \"kubernetes.io/projected/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-token\") pod \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") "
Nov 02 18:11:17.686303 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685298    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32802945-5e87-4339-a7eb-33212aab7ba5-kube-api-access-qbr2n\" (UniqueName: \"kubernetes.io/projected/32802945-5e87-4339-a7eb-33212aab7ba5-kube-api-access-qbr2n\") pod \"32802945-5e87-4339-a7eb-33212aab7ba5\" (UID: \"32802945-5e87-4339-a7eb-33212aab7ba5\") "
Nov 02 18:11:17.686303 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685327    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-workload-socket\") pod \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") "
Nov 02 18:11:17.686303 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685365    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume\") pod \"7e09749e-e648-40a3-a5f1-f0dc35fccf24\" (UID: \"7e09749e-e648-40a3-a5f1-f0dc35fccf24\") "
Nov 02 18:11:17.686503 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685398    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8c60dfaa-dabe-4726-91b5-165443ff3971-kube-api-access-qr8hh\" (UniqueName: \"kubernetes.io/projected/8c60dfaa-dabe-4726-91b5-165443ff3971-kube-api-access-qr8hh\") pod \"8c60dfaa-dabe-4726-91b5-165443ff3971\" (UID: \"8c60dfaa-dabe-4726-91b5-165443ff3971\") "
Nov 02 18:11:17.686503 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685431    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-istio-data\") pod \"551602c4-254b-4532-8961-a98bf0f81089\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") "
Nov 02 18:11:17.686503 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685454    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-istio-envoy\") pod \"551602c4-254b-4532-8961-a98bf0f81089\" (UID: \"551602c4-254b-4532-8961-a98bf0f81089\") "
Nov 02 18:11:17.686503 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685482    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume\") pod \"cf85b137-2b1e-4e22-8f6e-744a02ea4e46\" (UID: \"cf85b137-2b1e-4e22-8f6e-744a02ea4e46\") "
Nov 02 18:11:17.686503 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685502    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-credential-socket\") pod \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") "
Nov 02 18:11:17.686702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685537    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-token\" (UniqueName: \"kubernetes.io/projected/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-token\") pod \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") "
Nov 02 18:11:17.686702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685578    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istiod-ca-cert\") pod \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\" (UID: \"67b961eb-ab19-4d8d-bb2a-db7c94a93436\") "
Nov 02 18:11:17.686702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685627    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0996b2fa-40e8-469c-9ace-75458691fa25-kube-api-access-cx2dz\" (UniqueName: \"kubernetes.io/projected/0996b2fa-40e8-469c-9ace-75458691fa25-kube-api-access-cx2dz\") pod \"0996b2fa-40e8-469c-9ace-75458691fa25\" (UID: \"0996b2fa-40e8-469c-9ace-75458691fa25\") "
Nov 02 18:11:17.686702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685648    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-data\") pod \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\" (UID: \"3c8d9150-66d1-4be8-8bca-3782fb0eac36\") "
Nov 02 18:11:17.687131 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.687105    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbrdf\" (UniqueName: \"kubernetes.io/projected/44b66f41-99ae-446b-a50a-2ec0b71563a4-kube-api-access-qbrdf\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.687268 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.687250    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f7bnx\" (UniqueName: \"kubernetes.io/projected/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-kube-api-access-f7bnx\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.687362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.687349    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.687444 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.687431    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.687521 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.687510    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.687604 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.687593    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k85vn\" (UniqueName: \"kubernetes.io/projected/22b556c1-112b-49d4-932c-0143e912a54e-kube-api-access-k85vn\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.687681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.687669    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-crv8x\" (UniqueName: \"kubernetes.io/projected/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-kube-api-access-crv8x\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.687760 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.687748    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gv7wn\" (UniqueName: \"kubernetes.io/projected/82ba53b4-e91a-4299-9498-f5abcd34576b-kube-api-access-gv7wn\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.687872 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.687858    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rf96w\" (UniqueName: \"kubernetes.io/projected/3d96e066-10c0-4fe2-9e24-ffd6195b22af-kube-api-access-rf96w\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.687948 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.687935    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/44b66f41-99ae-446b-a50a-2ec0b71563a4-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.688014 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.688006    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.688071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.688057    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.688130 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.688074    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.688130 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.688088    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dpl8s\" (UniqueName: \"kubernetes.io/projected/3d398f17-735a-4d4b-b1cc-5b24fd266535-kube-api-access-dpl8s\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.688130 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684088    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-istio-data" pod "d4161075-440d-4373-b878-57c77612c6d2" (UID: "d4161075-440d-4373-b878-57c77612c6d2"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.688130 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684301    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-workload-certs" pod "551602c4-254b-4532-8961-a98bf0f81089" (UID: "551602c4-254b-4532-8961-a98bf0f81089"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.688130 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684495    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-credential-socket" pod "67b961eb-ab19-4d8d-bb2a-db7c94a93436" (UID: "67b961eb-ab19-4d8d-bb2a-db7c94a93436"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.688354 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.684696    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-workload-certs" pod "32802945-5e87-4339-a7eb-33212aab7ba5" (UID: "32802945-5e87-4339-a7eb-33212aab7ba5"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.688354 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.685578    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4161075-440d-4373-b878-57c77612c6d2-istiod-ca-cert" pod "d4161075-440d-4373-b878-57c77612c6d2" (UID: "d4161075-440d-4373-b878-57c77612c6d2"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.688354 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.688103    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/82ba53b4-e91a-4299-9498-f5abcd34576b-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.688354 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.688193    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/82ba53b4-e91a-4299-9498-f5abcd34576b-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.688354 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.688211    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d96e066-10c0-4fe2-9e24-ffd6195b22af-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.688354 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.688227    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e141d845-db9d-4f94-b05e-cfb7848e8fef-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.688354 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.688255    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/22b556c1-112b-49d4-932c-0143e912a54e-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.688681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.688269    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p9t7z\" (UniqueName: \"kubernetes.io/projected/e141d845-db9d-4f94-b05e-cfb7848e8fef-kube-api-access-p9t7z\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.688681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.688417    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.688681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.688432    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/f4ee5c50-5738-496a-b5ef-c86ead76e0f6-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.688681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.688446    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/3d398f17-735a-4d4b-b1cc-5b24fd266535-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.688681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.688356    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-data" pod "3c8d9150-66d1-4be8-8bca-3782fb0eac36" (UID: "3c8d9150-66d1-4be8-8bca-3782fb0eac36"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.688957 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.688727    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-istio-data" pod "32802945-5e87-4339-a7eb-33212aab7ba5" (UID: "32802945-5e87-4339-a7eb-33212aab7ba5"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.690524 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.690326    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-workload-certs" pod "0996b2fa-40e8-469c-9ace-75458691fa25" (UID: "0996b2fa-40e8-469c-9ace-75458691fa25"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.690881 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.690674    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-workload-certs" pod "d4161075-440d-4373-b878-57c77612c6d2" (UID: "d4161075-440d-4373-b878-57c77612c6d2"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.692843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.692059    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32802945-5e87-4339-a7eb-33212aab7ba5-istiod-ca-cert" pod "32802945-5e87-4339-a7eb-33212aab7ba5" (UID: "32802945-5e87-4339-a7eb-33212aab7ba5"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.692843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.692132    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-workload-socket" pod "551602c4-254b-4532-8961-a98bf0f81089" (UID: "551602c4-254b-4532-8961-a98bf0f81089"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.695408 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.695325    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-credential-socket" pod "3c8d9150-66d1-4be8-8bca-3782fb0eac36" (UID: "3c8d9150-66d1-4be8-8bca-3782fb0eac36"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.697072 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.696806    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istiod-ca-cert" pod "67b961eb-ab19-4d8d-bb2a-db7c94a93436" (UID: "67b961eb-ab19-4d8d-bb2a-db7c94a93436"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.698870 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.698693    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-workload-socket" pod "32802945-5e87-4339-a7eb-33212aab7ba5" (UID: "32802945-5e87-4339-a7eb-33212aab7ba5"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.698870 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.698704    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-workload-socket" pod "67b961eb-ab19-4d8d-bb2a-db7c94a93436" (UID: "67b961eb-ab19-4d8d-bb2a-db7c94a93436"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.699671 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.699537    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istiod-ca-cert" pod "3c8d9150-66d1-4be8-8bca-3782fb0eac36" (UID: "3c8d9150-66d1-4be8-8bca-3782fb0eac36"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.699671 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.699638    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-istio-envoy" pod "32802945-5e87-4339-a7eb-33212aab7ba5" (UID: "32802945-5e87-4339-a7eb-33212aab7ba5"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.700064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.700040    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-credential-socket" pod "32802945-5e87-4339-a7eb-33212aab7ba5" (UID: "32802945-5e87-4339-a7eb-33212aab7ba5"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.700732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.700375    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-workload-socket" pod "0996b2fa-40e8-469c-9ace-75458691fa25" (UID: "0996b2fa-40e8-469c-9ace-75458691fa25"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.700732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.700510    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume" pod "cf85b137-2b1e-4e22-8f6e-744a02ea4e46" (UID: "cf85b137-2b1e-4e22-8f6e-744a02ea4e46"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.701537 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.701512    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume" pod "7e09749e-e648-40a3-a5f1-f0dc35fccf24" (UID: "7e09749e-e648-40a3-a5f1-f0dc35fccf24"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.702396 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.702151    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-istio-data" pod "551602c4-254b-4532-8961-a98bf0f81089" (UID: "551602c4-254b-4532-8961-a98bf0f81089"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.702589 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.702556    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-istio-data" pod "0996b2fa-40e8-469c-9ace-75458691fa25" (UID: "0996b2fa-40e8-469c-9ace-75458691fa25"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.702829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.702797    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-podinfo" pod "67b961eb-ab19-4d8d-bb2a-db7c94a93436" (UID: "67b961eb-ab19-4d8d-bb2a-db7c94a93436"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:17.706451 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.703235    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-credential-socket" pod "551602c4-254b-4532-8961-a98bf0f81089" (UID: "551602c4-254b-4532-8961-a98bf0f81089"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.706451 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.703459    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume" pod "1c190514-ba33-4844-a9ec-e4bb47b5f706" (UID: "1c190514-ba33-4844-a9ec-e4bb47b5f706"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.706451 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.703546    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/0996b2fa-40e8-469c-9ace-75458691fa25-istio-podinfo" pod "0996b2fa-40e8-469c-9ace-75458691fa25" (UID: "0996b2fa-40e8-469c-9ace-75458691fa25"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:17.706451 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.705447    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b961eb-ab19-4d8d-bb2a-db7c94a93436-kube-api-access-6vtvv" pod "67b961eb-ab19-4d8d-bb2a-db7c94a93436" (UID: "67b961eb-ab19-4d8d-bb2a-db7c94a93436"). InnerVolumeSpecName "kube-api-access-6vtvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.706451 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.705499    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-token" pod "3c8d9150-66d1-4be8-8bca-3782fb0eac36" (UID: "3c8d9150-66d1-4be8-8bca-3782fb0eac36"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.706719 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.705759    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-istio-envoy" pod "551602c4-254b-4532-8961-a98bf0f81089" (UID: "551602c4-254b-4532-8961-a98bf0f81089"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.707676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.706803    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-workload-certs" pod "67b961eb-ab19-4d8d-bb2a-db7c94a93436" (UID: "67b961eb-ab19-4d8d-bb2a-db7c94a93436"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.707676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.707051    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-credential-socket" pod "d4161075-440d-4373-b878-57c77612c6d2" (UID: "d4161075-440d-4373-b878-57c77612c6d2"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.707676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.707395    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume" pod "8c60dfaa-dabe-4726-91b5-165443ff3971" (UID: "8c60dfaa-dabe-4726-91b5-165443ff3971"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.707676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.707417    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/551602c4-254b-4532-8961-a98bf0f81089-istiod-ca-cert" pod "551602c4-254b-4532-8961-a98bf0f81089" (UID: "551602c4-254b-4532-8961-a98bf0f81089"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.709679 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.707965    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-istio-envoy" pod "0996b2fa-40e8-469c-9ace-75458691fa25" (UID: "0996b2fa-40e8-469c-9ace-75458691fa25"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.709679 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.708079    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32802945-5e87-4339-a7eb-33212aab7ba5-kube-api-access-qbr2n" pod "32802945-5e87-4339-a7eb-33212aab7ba5" (UID: "32802945-5e87-4339-a7eb-33212aab7ba5"). InnerVolumeSpecName "kube-api-access-qbr2n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.709679 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.708142    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32802945-5e87-4339-a7eb-33212aab7ba5-istio-token" pod "32802945-5e87-4339-a7eb-33212aab7ba5" (UID: "32802945-5e87-4339-a7eb-33212aab7ba5"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.709679 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.708176    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0996b2fa-40e8-469c-9ace-75458691fa25-istiod-ca-cert" pod "0996b2fa-40e8-469c-9ace-75458691fa25" (UID: "0996b2fa-40e8-469c-9ace-75458691fa25"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:17.709679 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.708991    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0996b2fa-40e8-469c-9ace-75458691fa25-kube-api-access-cx2dz" pod "0996b2fa-40e8-469c-9ace-75458691fa25" (UID: "0996b2fa-40e8-469c-9ace-75458691fa25"). InnerVolumeSpecName "kube-api-access-cx2dz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.710293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.709241    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-workload-certs" pod "3c8d9150-66d1-4be8-8bca-3782fb0eac36" (UID: "3c8d9150-66d1-4be8-8bca-3782fb0eac36"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.710293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.709275    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1c190514-ba33-4844-a9ec-e4bb47b5f706-kube-api-access-b4g6z" pod "1c190514-ba33-4844-a9ec-e4bb47b5f706" (UID: "1c190514-ba33-4844-a9ec-e4bb47b5f706"). InnerVolumeSpecName "kube-api-access-b4g6z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.711307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.710569    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551602c4-254b-4532-8961-a98bf0f81089-istio-token" pod "551602c4-254b-4532-8961-a98bf0f81089" (UID: "551602c4-254b-4532-8961-a98bf0f81089"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.711307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.710575    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-istio-envoy" pod "d4161075-440d-4373-b878-57c77612c6d2" (UID: "d4161075-440d-4373-b878-57c77612c6d2"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.711307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.710633    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-workload-socket" pod "d4161075-440d-4373-b878-57c77612c6d2" (UID: "d4161075-440d-4373-b878-57c77612c6d2"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.711307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.710665    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-data" pod "67b961eb-ab19-4d8d-bb2a-db7c94a93436" (UID: "67b961eb-ab19-4d8d-bb2a-db7c94a93436"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.711307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.710700    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-credential-socket" pod "0996b2fa-40e8-469c-9ace-75458691fa25" (UID: "0996b2fa-40e8-469c-9ace-75458691fa25"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.711577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.710761    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-kube-api-access-9w876" pod "cf85b137-2b1e-4e22-8f6e-744a02ea4e46" (UID: "cf85b137-2b1e-4e22-8f6e-744a02ea4e46"). InnerVolumeSpecName "kube-api-access-9w876". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.711577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.711271    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-workload-socket" pod "3c8d9150-66d1-4be8-8bca-3782fb0eac36" (UID: "3c8d9150-66d1-4be8-8bca-3782fb0eac36"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.713280 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.713255    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-envoy" pod "3c8d9150-66d1-4be8-8bca-3782fb0eac36" (UID: "3c8d9150-66d1-4be8-8bca-3782fb0eac36"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.713738 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.713619    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3c8d9150-66d1-4be8-8bca-3782fb0eac36-kube-api-access-2vktq" pod "3c8d9150-66d1-4be8-8bca-3782fb0eac36" (UID: "3c8d9150-66d1-4be8-8bca-3782fb0eac36"). InnerVolumeSpecName "kube-api-access-2vktq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.713738 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.713707    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0996b2fa-40e8-469c-9ace-75458691fa25-istio-token" pod "0996b2fa-40e8-469c-9ace-75458691fa25" (UID: "0996b2fa-40e8-469c-9ace-75458691fa25"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.713963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.713935    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/d4161075-440d-4373-b878-57c77612c6d2-istio-podinfo" pod "d4161075-440d-4373-b878-57c77612c6d2" (UID: "d4161075-440d-4373-b878-57c77612c6d2"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:17.714032 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.714022    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/551602c4-254b-4532-8961-a98bf0f81089-kube-api-access-z2dvq" pod "551602c4-254b-4532-8961-a98bf0f81089" (UID: "551602c4-254b-4532-8961-a98bf0f81089"). InnerVolumeSpecName "kube-api-access-z2dvq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.714495 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.714466    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/551602c4-254b-4532-8961-a98bf0f81089-istio-podinfo" pod "551602c4-254b-4532-8961-a98bf0f81089" (UID: "551602c4-254b-4532-8961-a98bf0f81089"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:17.714687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.714663    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8c60dfaa-dabe-4726-91b5-165443ff3971-kube-api-access-qr8hh" pod "8c60dfaa-dabe-4726-91b5-165443ff3971" (UID: "8c60dfaa-dabe-4726-91b5-165443ff3971"). InnerVolumeSpecName "kube-api-access-qr8hh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.715145 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.715030    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/32802945-5e87-4339-a7eb-33212aab7ba5-istio-podinfo" pod "32802945-5e87-4339-a7eb-33212aab7ba5" (UID: "32802945-5e87-4339-a7eb-33212aab7ba5"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:17.718639 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.718608    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-envoy" pod "67b961eb-ab19-4d8d-bb2a-db7c94a93436" (UID: "67b961eb-ab19-4d8d-bb2a-db7c94a93436"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:17.718639 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.718617    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-token" pod "67b961eb-ab19-4d8d-bb2a-db7c94a93436" (UID: "67b961eb-ab19-4d8d-bb2a-db7c94a93436"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.718743 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.718639    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e09749e-e648-40a3-a5f1-f0dc35fccf24-kube-api-access-f586g" pod "7e09749e-e648-40a3-a5f1-f0dc35fccf24" (UID: "7e09749e-e648-40a3-a5f1-f0dc35fccf24"). InnerVolumeSpecName "kube-api-access-f586g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.719210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.719159    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4161075-440d-4373-b878-57c77612c6d2-istio-token" pod "d4161075-440d-4373-b878-57c77612c6d2" (UID: "d4161075-440d-4373-b878-57c77612c6d2"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.719398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.719369    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-podinfo" pod "3c8d9150-66d1-4be8-8bca-3782fb0eac36" (UID: "3c8d9150-66d1-4be8-8bca-3782fb0eac36"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:17.719661 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.719635    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4161075-440d-4373-b878-57c77612c6d2-kube-api-access-grksl" pod "d4161075-440d-4373-b878-57c77612c6d2" (UID: "d4161075-440d-4373-b878-57c77612c6d2"). InnerVolumeSpecName "kube-api-access-grksl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:17.790163 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790072    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/d4161075-440d-4373-b878-57c77612c6d2-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790163 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790115    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8c60dfaa-dabe-4726-91b5-165443ff3971-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790163 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790126    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-grksl\" (UniqueName: \"kubernetes.io/projected/d4161075-440d-4373-b878-57c77612c6d2-kube-api-access-grksl\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790163 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790136    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790163 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790153    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/32802945-5e87-4339-a7eb-33212aab7ba5-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790163 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790162    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790163 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790170    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790178    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790187    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/551602c4-254b-4532-8961-a98bf0f81089-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790194    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790202    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790212    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790220    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0996b2fa-40e8-469c-9ace-75458691fa25-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790228    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/1c190514-ba33-4844-a9ec-e4bb47b5f706-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790238    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2vktq\" (UniqueName: \"kubernetes.io/projected/3c8d9150-66d1-4be8-8bca-3782fb0eac36-kube-api-access-2vktq\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790250    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/551602c4-254b-4532-8961-a98bf0f81089-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790260    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790268    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790275    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qbr2n\" (UniqueName: \"kubernetes.io/projected/32802945-5e87-4339-a7eb-33212aab7ba5-kube-api-access-qbr2n\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790304    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.790998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790317    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/7e09749e-e648-40a3-a5f1-f0dc35fccf24-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790328    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qr8hh\" (UniqueName: \"kubernetes.io/projected/8c60dfaa-dabe-4726-91b5-165443ff3971-kube-api-access-qr8hh\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790336    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790346    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790354    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790362    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790372    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790380    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790387    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cx2dz\" (UniqueName: \"kubernetes.io/projected/0996b2fa-40e8-469c-9ace-75458691fa25-kube-api-access-cx2dz\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790395    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790403    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b4g6z\" (UniqueName: \"kubernetes.io/projected/1c190514-ba33-4844-a9ec-e4bb47b5f706-kube-api-access-b4g6z\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790411    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790419    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790427    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790434    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/d4161075-440d-4373-b878-57c77612c6d2-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790444    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790451    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790458    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790465    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/32802945-5e87-4339-a7eb-33212aab7ba5-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790472    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/32802945-5e87-4339-a7eb-33212aab7ba5-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790483    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/0996b2fa-40e8-469c-9ace-75458691fa25-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790490    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z2dvq\" (UniqueName: \"kubernetes.io/projected/551602c4-254b-4532-8961-a98bf0f81089-kube-api-access-z2dvq\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790501    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.791935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790509    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/551602c4-254b-4532-8961-a98bf0f81089-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790517    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/d4161075-440d-4373-b878-57c77612c6d2-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790525    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790532    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790539    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/67b961eb-ab19-4d8d-bb2a-db7c94a93436-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790546    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/32802945-5e87-4339-a7eb-33212aab7ba5-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790553    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790561    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6vtvv\" (UniqueName: \"kubernetes.io/projected/67b961eb-ab19-4d8d-bb2a-db7c94a93436-kube-api-access-6vtvv\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790569    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790576    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790584    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9w876\" (UniqueName: \"kubernetes.io/projected/cf85b137-2b1e-4e22-8f6e-744a02ea4e46-kube-api-access-9w876\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790591    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/0996b2fa-40e8-469c-9ace-75458691fa25-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790599    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/551602c4-254b-4532-8961-a98bf0f81089-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790607    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/3c8d9150-66d1-4be8-8bca-3782fb0eac36-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790615    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/67b961eb-ab19-4d8d-bb2a-db7c94a93436-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790622    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792930 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790630    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/0996b2fa-40e8-469c-9ace-75458691fa25-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792930 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790637    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/d4161075-440d-4373-b878-57c77612c6d2-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:17.792930 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:17.790646    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f586g\" (UniqueName: \"kubernetes.io/projected/7e09749e-e648-40a3-a5f1-f0dc35fccf24-kube-api-access-f586g\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:18.450175 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.450136    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-console/downloads-6dd789c8b7-pgmzg"]
Nov 02 18:11:18.455345 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.455319    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-console/downloads-6dd789c8b7-pgmzg"]
Nov 02 18:11:18.484728 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.484704    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"
Nov 02 18:11:18.484905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.484728    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g" event={"ID":"44b66f41-99ae-446b-a50a-2ec0b71563a4","Type":"ContainerDied","Data":"af7639a1c401b49f95ce04b0dc0df7c785bd5660b007f844f58d72990f9b9823"}
Nov 02 18:11:18.484905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.484836    4456 scope.go:122] "RemoveContainer" containerID="a15fc9b95bc579e3250f894c650e5c31574a16f7a9e567db93f2291e2e8e8b0a"
Nov 02 18:11:18.487077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.486493    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"
Nov 02 18:11:18.487077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.486510    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn" event={"ID":"3d96e066-10c0-4fe2-9e24-ffd6195b22af","Type":"ContainerDied","Data":"d932a52666d4957f9e059f45c9de22ea0c348c2deebf0b105adbc40857774316"}
Nov 02 18:11:18.489734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.489714    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"
Nov 02 18:11:18.490097 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.489722    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb" event={"ID":"cf85b137-2b1e-4e22-8f6e-744a02ea4e46","Type":"ContainerDied","Data":"1309254054a37595d00abffd63c04c21ea5ae37f9a2f91210c745674592bb4be"}
Nov 02 18:11:18.491724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.491705    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"
Nov 02 18:11:18.491831 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.491709    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r" event={"ID":"67b961eb-ab19-4d8d-bb2a-db7c94a93436","Type":"ContainerDied","Data":"42ee0fddf775b9d2d49f7e881c391a9dbd7969d720ee7c45060a8dbe249c0fae"}
Nov 02 18:11:18.493620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.493601    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"
Nov 02 18:11:18.494694 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.493611    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg" event={"ID":"96db7b02-b878-435a-869e-1d30e147c61d","Type":"ContainerDied","Data":"cdc808bf198ef177770722cadcded05a71fa080190c7dd200fb6f2ee77c03ba2"}
Nov 02 18:11:18.495944 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.495925    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"
Nov 02 18:11:18.496491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.496057    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb" event={"ID":"551602c4-254b-4532-8961-a98bf0f81089","Type":"ContainerDied","Data":"39a7fb1949a9a5d2645eaa5f04aa88de3c666d66cd68aace8fb45d834bbc78d3"}
Nov 02 18:11:18.497985 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.497958    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z" event={"ID":"d4161075-440d-4373-b878-57c77612c6d2","Type":"ContainerDied","Data":"67a56185b766be865af25189d9f16bedea44d4a32a57cfd7c72c6d0fc82578e3"}
Nov 02 18:11:18.498093 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.498060    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"
Nov 02 18:11:18.502585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.502001    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6" event={"ID":"1c190514-ba33-4844-a9ec-e4bb47b5f706","Type":"ContainerDied","Data":"e2874138d3aa9323123ddec8ad7186f666235e1a9186836d7e0aed34dc66cb17"}
Nov 02 18:11:18.502585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.502135    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"
Nov 02 18:11:18.504966 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.503757    4456 scope.go:122] "RemoveContainer" containerID="4fa143115bc295dd3e06ce938045b94490edee81d1b9aa59b048d0f8a259384b"
Nov 02 18:11:18.516033 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.515998    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5" event={"ID":"0996b2fa-40e8-469c-9ace-75458691fa25","Type":"ContainerDied","Data":"bdd3bb397d2d235caac6e3e217c612baf7d6db4d168a86f42bc1ab293dc8256d"}
Nov 02 18:11:18.516513 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.516493    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"
Nov 02 18:11:18.521491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.521441    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89" event={"ID":"8c60dfaa-dabe-4726-91b5-165443ff3971","Type":"ContainerDied","Data":"5f90ce0fb09944493d00fc7e0feface8077779a655da9071d0560154b6550598"}
Nov 02 18:11:18.521491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.521563    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"
Nov 02 18:11:18.527637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.527613    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"
Nov 02 18:11:18.529041 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.528433    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr" event={"ID":"f4ee5c50-5738-496a-b5ef-c86ead76e0f6","Type":"ContainerDied","Data":"877445f9d12eef4c52383326ec19ff97cf66c0b2f107c3c3736b9e41fcd66057"}
Nov 02 18:11:18.532489 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.532462    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45" event={"ID":"7e09749e-e648-40a3-a5f1-f0dc35fccf24","Type":"ContainerDied","Data":"27834b12d06e3c3b988ff0cd49935574491e3c10b89279f045db0b5cce79e9b9"}
Nov 02 18:11:18.532585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.532574    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"
Nov 02 18:11:18.537827 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.537786    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr" event={"ID":"e141d845-db9d-4f94-b05e-cfb7848e8fef","Type":"ContainerDied","Data":"fd67a031eae7babedd49e82fb55f1afbb77cefc2cfe2f850b85471356a865190"}
Nov 02 18:11:18.537926 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.537909    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"
Nov 02 18:11:18.539865 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.539837    4456 scope.go:122] "RemoveContainer" containerID="1f50936a9578f37077e2da181b984035ceacd77c575058ecec4a61436289a702"
Nov 02 18:11:18.541555 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.541532    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr" event={"ID":"22b556c1-112b-49d4-932c-0143e912a54e","Type":"ContainerDied","Data":"73ad40a2e08e2a89f3f3a65b0278695bf804ddfca38bfd4dc0b160b370ad7f6a"}
Nov 02 18:11:18.541785 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.541765    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"
Nov 02 18:11:18.544709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.544310    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"
Nov 02 18:11:18.544709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.544332    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5" event={"ID":"3d398f17-735a-4d4b-b1cc-5b24fd266535","Type":"ContainerDied","Data":"772c1ae8cdded18a6da829cad4beb7330175fc9c81b6e9f656c5a3b80448a335"}
Nov 02 18:11:18.546394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.546284    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6" event={"ID":"60c0e838-5c79-4bf7-a7c6-87903ae8bd8c","Type":"ContainerDied","Data":"c6c35dcca3f6cb5573b9cede13b9a4db8e96921ce5c70e83fd46b5db3efda2a2"}
Nov 02 18:11:18.546394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.546303    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"
Nov 02 18:11:18.548875 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.548012    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n" event={"ID":"85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3","Type":"ContainerDied","Data":"d719284df40bfeba040610251ea9252cec8432f5ba7b2b4f480accefeb3949d6"}
Nov 02 18:11:18.548875 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.548014    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"
Nov 02 18:11:18.553345 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.553228    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"
Nov 02 18:11:18.554973 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.553023    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl" event={"ID":"3c8d9150-66d1-4be8-8bca-3782fb0eac36","Type":"ContainerDied","Data":"db13028a213d83d2cc7de22049d0661e4b58c919befb6899b5eafb0bc7b2f632"}
Nov 02 18:11:18.568879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.562397    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw" event={"ID":"32802945-5e87-4339-a7eb-33212aab7ba5","Type":"ContainerDied","Data":"986adc966a1f428a26ef1fcc38a47dded87dad6c3c54e2b6d931b2171d229419"}
Nov 02 18:11:18.568879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.562503    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"
Nov 02 18:11:18.568879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.567486    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"
Nov 02 18:11:18.572114 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.569480    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"
Nov 02 18:11:18.572114 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.569887    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt" event={"ID":"82ba53b4-e91a-4299-9498-f5abcd34576b","Type":"ContainerDied","Data":"34501ef515064b29330a7fee7503fbc73cfb72b427115992b8ff7db7323b0c24"}
Nov 02 18:11:18.577014 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.576988    4456 scope.go:122] "RemoveContainer" containerID="f5f583c97bede29bd199be0ec2003a50248abd83bffd0425ce6e7cb495a11ca3"
Nov 02 18:11:18.593666 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.593646    4456 scope.go:122] "RemoveContainer" containerID="1780545d23b0db3b73d9b9c93d4463de2ecb228acbbac446492792422b835d1c"
Nov 02 18:11:18.608634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.608244    4456 scope.go:122] "RemoveContainer" containerID="6a3d5cd7a3cf30d356953f12d067b1b473d9827087fa6525e30fb999560de281"
Nov 02 18:11:18.626415 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.626394    4456 scope.go:122] "RemoveContainer" containerID="e079ea8bc432c6b15221e1f0f0d1f1094347bece3a5dc356bdb25b749c2da4d5"
Nov 02 18:11:18.640772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.640739    4456 scope.go:122] "RemoveContainer" containerID="e8fac6b06137604d986996a69eb0437546b5e8e988303a1180286636239ba239"
Nov 02 18:11:18.653910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.653886    4456 scope.go:122] "RemoveContainer" containerID="0f61b587f093bc2a10aa5b15cd1853e4adc293fe1098512029b4141d26f10d42"
Nov 02 18:11:18.674517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.674485    4456 scope.go:122] "RemoveContainer" containerID="47ce80eead8cafb5ad533cbe8d61db652959180c5d1b75fef0c82fd03c99ce2f"
Nov 02 18:11:18.688458 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.688430    4456 scope.go:122] "RemoveContainer" containerID="52131bd7b83ca9822e72e558a3dff218d4a6e09dcbe0ddc00be3cd9e069c1cbb"
Nov 02 18:11:18.701577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.701552    4456 scope.go:122] "RemoveContainer" containerID="8c6acffecf9ac35beb62fec780633f1baf430b68dfeefab50a43468a043033a3"
Nov 02 18:11:18.714372 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.714344    4456 scope.go:122] "RemoveContainer" containerID="723c09d5d9658eb778a71667f74238ad51cacca740ec11432caf9b15357688e9"
Nov 02 18:11:18.733620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.733597    4456 scope.go:122] "RemoveContainer" containerID="968e2de3ae2524b2141ebfc9d2cc91b4232330bb2b1e8308c1e4e1329d68b932"
Nov 02 18:11:18.748928 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.748907    4456 scope.go:122] "RemoveContainer" containerID="6c24194466a92255caca64b6c1b7d27bee17e48cdd5018b681fa9f4ecb1b733e"
Nov 02 18:11:18.764629 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.764596    4456 scope.go:122] "RemoveContainer" containerID="5eca9c03a08e76e5e6f070cb8052b78663a3818d87186a38c4402f7bf5034995"
Nov 02 18:11:18.779264 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.779126    4456 scope.go:122] "RemoveContainer" containerID="83a5c5da51e2a33c43fedd46148933222917fbcba3a4cacd320bc29364bbf103"
Nov 02 18:11:18.797017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.796989    4456 scope.go:122] "RemoveContainer" containerID="7b5ffd17510b581a401cae0b3017c71076c1341b3ae93eed0e79ed9040740ec5"
Nov 02 18:11:18.812732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.812709    4456 scope.go:122] "RemoveContainer" containerID="cae3f46a8bc87e7369f5ef014d19a5821ecaa21b04f97dd5ed71d6ee6eb99acf"
Nov 02 18:11:18.824923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:18.824891    4456 scope.go:122] "RemoveContainer" containerID="e0bbfd6f9c197b1322c631c19d97f927c7c4b56631d9cd54e8dd5474d3b87b5d"
Nov 02 18:11:19.052442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.052248    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:19.511743 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.511708    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"]
Nov 02 18:11:19.519142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.519111    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-bb6d2e-95695bd75-f66qr"]
Nov 02 18:11:19.552614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.552580    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"]
Nov 02 18:11:19.558062 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.558035    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-6r66r"]
Nov 02 18:11:19.599850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.599806    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"]
Nov 02 18:11:19.616254 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.616222    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" path="/var/lib/kubelet/pods/67b961eb-ab19-4d8d-bb2a-db7c94a93436/volumes"
Nov 02 18:11:19.616773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.616756    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" path="/var/lib/kubelet/pods/6e78d6c7-c325-48f2-af90-804dade84d8f/volumes"
Nov 02 18:11:19.617244 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.617229    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" path="/var/lib/kubelet/pods/f4ee5c50-5738-496a-b5ef-c86ead76e0f6/volumes"
Nov 02 18:11:19.617920 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.617905    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a29999-openshift-default-6dcbf59465-2z6wt"]
Nov 02 18:11:19.649501 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.649474    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"]
Nov 02 18:11:19.654672 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.654645    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-a5c1da-openshift-default-86f5db6fb9-fm87z"]
Nov 02 18:11:19.696017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.695987    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"]
Nov 02 18:11:19.700770 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.700744    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-jkmn5"]
Nov 02 18:11:19.735654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.735622    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"]
Nov 02 18:11:19.740405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.740381    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-v5qhw"]
Nov 02 18:11:19.777785 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.777697    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/thanos-querier-667db95b96-kmwrg"
Nov 02 18:11:19.777785 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.777769    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"]
Nov 02 18:11:19.782350 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.782310    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-872354-dbfcb5b87-n4r4g"]
Nov 02 18:11:19.825546 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.825512    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"]
Nov 02 18:11:19.830007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.829978    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-872354-openshift-default-59df9df7-vnmfl"]
Nov 02 18:11:19.870762 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.870733    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"]
Nov 02 18:11:19.880910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.880885    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-a-96768b-openshift-default-547568cbcf-h5lbb"]
Nov 02 18:11:19.937072 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.937039    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"]
Nov 02 18:11:19.941568 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.941533    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-fbbbf8-54c78f96dc-mrgd5"]
Nov 02 18:11:19.980646 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.980606    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"]
Nov 02 18:11:19.984948 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:19.984924    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-b-2c9b1d-99f47c5f5-bnzqn"]
Nov 02 18:11:20.027198 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.027168    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"]
Nov 02 18:11:20.035142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.035078    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-96768b-5c898c47d9-8957f"]
Nov 02 18:11:20.074742 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.074717    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"]
Nov 02 18:11:20.082040 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.082015    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a29999-fbdcd7fb7-mcv45"]
Nov 02 18:11:20.114268 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.114226    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"]
Nov 02 18:11:20.118414 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.118391    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a29999-6f749d4d55-5qpv6"]
Nov 02 18:11:20.154619 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.154576    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"]
Nov 02 18:11:20.159104 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.159080    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-d5b905-6754d44cfc-jjlkr"]
Nov 02 18:11:20.194755 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.194717    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"]
Nov 02 18:11:20.198950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.198908    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-2c9b1d-7f849b8d65-s9mz6"]
Nov 02 18:11:20.238095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.238061    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"]
Nov 02 18:11:20.242993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.242966    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-96768b-587d5d665f-v78lr"]
Nov 02 18:11:20.272800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.272771    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"]
Nov 02 18:11:20.280445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.280416    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-fbbbf8-6b9c86bf5-cxq6n"]
Nov 02 18:11:20.313931 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.313842    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"]
Nov 02 18:11:20.320912 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.320886    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-inf-a-a5c1da-7494d7cdb7-tbqrg"]
Nov 02 18:11:20.358388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.358361    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"]
Nov 02 18:11:20.363278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.363254    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-a5c1da-69ff45468c-p5w89"]
Nov 02 18:11:20.401923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.401887    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"]
Nov 02 18:11:20.408385 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.408358    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-a-872354-dcf6d859f-bcdtb"]
Nov 02 18:11:20.800477 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.800448    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 18:11:20.801106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:20.801085    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 18:11:21.056683 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.056585    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:21.617855 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.617804    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" path="/var/lib/kubelet/pods/0996b2fa-40e8-469c-9ace-75458691fa25/volumes"
Nov 02 18:11:21.619013 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.618988    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" path="/var/lib/kubelet/pods/1c190514-ba33-4844-a9ec-e4bb47b5f706/volumes"
Nov 02 18:11:21.619408 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.619394    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="22b556c1-112b-49d4-932c-0143e912a54e" path="/var/lib/kubelet/pods/22b556c1-112b-49d4-932c-0143e912a54e/volumes"
Nov 02 18:11:21.620242 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.620219    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" path="/var/lib/kubelet/pods/32802945-5e87-4339-a7eb-33212aab7ba5/volumes"
Nov 02 18:11:21.620750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.620736    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" path="/var/lib/kubelet/pods/3c8d9150-66d1-4be8-8bca-3782fb0eac36/volumes"
Nov 02 18:11:21.621196 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.621182    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" path="/var/lib/kubelet/pods/3d398f17-735a-4d4b-b1cc-5b24fd266535/volumes"
Nov 02 18:11:21.622566 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.622551    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" path="/var/lib/kubelet/pods/3d96e066-10c0-4fe2-9e24-ffd6195b22af/volumes"
Nov 02 18:11:21.623012 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.622997    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="44b66f41-99ae-446b-a50a-2ec0b71563a4" path="/var/lib/kubelet/pods/44b66f41-99ae-446b-a50a-2ec0b71563a4/volumes"
Nov 02 18:11:21.623491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.623477    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="551602c4-254b-4532-8961-a98bf0f81089" path="/var/lib/kubelet/pods/551602c4-254b-4532-8961-a98bf0f81089/volumes"
Nov 02 18:11:21.624184 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.624165    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" path="/var/lib/kubelet/pods/5ce3335c-36e6-4018-a264-5b79085f09f3/volumes"
Nov 02 18:11:21.624606 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.624588    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" path="/var/lib/kubelet/pods/60c0e838-5c79-4bf7-a7c6-87903ae8bd8c/volumes"
Nov 02 18:11:21.625021 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.625005    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" path="/var/lib/kubelet/pods/7e09749e-e648-40a3-a5f1-f0dc35fccf24/volumes"
Nov 02 18:11:21.625482 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.625467    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" path="/var/lib/kubelet/pods/82ba53b4-e91a-4299-9498-f5abcd34576b/volumes"
Nov 02 18:11:21.626209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.626190    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" path="/var/lib/kubelet/pods/85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3/volumes"
Nov 02 18:11:21.626622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.626609    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" path="/var/lib/kubelet/pods/8c60dfaa-dabe-4726-91b5-165443ff3971/volumes"
Nov 02 18:11:21.627084 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.627071    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="96db7b02-b878-435a-869e-1d30e147c61d" path="/var/lib/kubelet/pods/96db7b02-b878-435a-869e-1d30e147c61d/volumes"
Nov 02 18:11:21.627609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.627595    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" path="/var/lib/kubelet/pods/cf85b137-2b1e-4e22-8f6e-744a02ea4e46/volumes"
Nov 02 18:11:21.628133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.628119    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4161075-440d-4373-b878-57c77612c6d2" path="/var/lib/kubelet/pods/d4161075-440d-4373-b878-57c77612c6d2/volumes"
Nov 02 18:11:21.628579 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:21.628566    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e141d845-db9d-4f94-b05e-cfb7848e8fef" path="/var/lib/kubelet/pods/e141d845-db9d-4f94-b05e-cfb7848e8fef/volumes"
Nov 02 18:11:22.481546 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:22.481518    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 18:11:22.607015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:22.606980    4456 scope.go:122] "RemoveContainer" containerID="d381788d1f97c945f12e6a0f9ca8aea3a3f0d92249a339bd95400d14c394ce40"
Nov 02 18:11:22.855401 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:22.855304    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:23.641798 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:23.641765    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="cert-manager/cert-manager-597b95cb84-hr8vz" event={"ID":"e73be583-2a26-4379-8169-0a6b8784a11b","Type":"ContainerStarted","Data":"f96cde46ce9e6bd8c85a1fe90bed09c0f77533bc2963aa7cd8ed95554bf33b1f"}
Nov 02 18:11:24.752878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:24.748196    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:24.810838 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:24.809223    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 18:11:24.923838 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:24.923083    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-jobset-operator/jobset-controller-manager-77cfb5f94-gfbr9"
Nov 02 18:11:26.781699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:26.781663    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:27.481259 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:27.481226    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="opendatahub-operator-system/opendatahub-operator-controller-manager-76f76b79c-7n462"
Nov 02 18:11:28.751339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:28.751306    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:30.726299 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:30.726258    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:32.677935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:32.677899    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:33.105605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:33.105555    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" podUID="0db36d92-4712-4338-958a-ebabf2c137ba" containerName="registry" containerID="cri-o://2501b584d5ee7323484f50ac5f32dd50c7fef23cf98bd9bf3120550c1d0985af" gracePeriod=30
Nov 02 18:11:33.764736 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:33.764705    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:34.427086 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.427052    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"]
Nov 02 18:11:34.428193 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428169    4456 cpu_manager.go:414] "containerMap: removing container" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy"
Nov 02 18:11:34.428234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428200    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy"
Nov 02 18:11:34.428234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428226    4456 cpu_manager.go:414] "containerMap: removing container" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerName="payload-processing"
Nov 02 18:11:34.428289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428235    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerName="payload-processing"
Nov 02 18:11:34.428289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428258    4456 cpu_manager.go:414] "containerMap: removing container" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy"
Nov 02 18:11:34.428289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428267    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy"
Nov 02 18:11:34.428289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428277    4456 cpu_manager.go:414] "containerMap: removing container" podUID="8064da0c-fc5d-493c-bcdf-4327716d4cc2" containerName="migrator"
Nov 02 18:11:34.428289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428285    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8064da0c-fc5d-493c-bcdf-4327716d4cc2" containerName="migrator"
Nov 02 18:11:34.428432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428308    4456 cpu_manager.go:414] "containerMap: removing container" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy"
Nov 02 18:11:34.428432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428315    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy"
Nov 02 18:11:34.428432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428332    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e141d845-db9d-4f94-b05e-cfb7848e8fef" containerName="payload-processing"
Nov 02 18:11:34.428432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428340    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e141d845-db9d-4f94-b05e-cfb7848e8fef" containerName="payload-processing"
Nov 02 18:11:34.428432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428354    4456 cpu_manager.go:414] "containerMap: removing container" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy"
Nov 02 18:11:34.428432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428361    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy"
Nov 02 18:11:34.428432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428372    4456 cpu_manager.go:414] "containerMap: removing container" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy"
Nov 02 18:11:34.428432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428380    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy"
Nov 02 18:11:34.428432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428401    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerName="payload-processing"
Nov 02 18:11:34.428432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428408    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerName="payload-processing"
Nov 02 18:11:34.428432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428418    4456 cpu_manager.go:414] "containerMap: removing container" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy"
Nov 02 18:11:34.428709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428425    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy"
Nov 02 18:11:34.428709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428438    4456 cpu_manager.go:414] "containerMap: removing container" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy"
Nov 02 18:11:34.428709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428446    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy"
Nov 02 18:11:34.428709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428463    4456 cpu_manager.go:414] "containerMap: removing container" podUID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" containerName="payload-pre-processing"
Nov 02 18:11:34.428709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428471    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" containerName="payload-pre-processing"
Nov 02 18:11:34.428709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428483    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy"
Nov 02 18:11:34.428709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428490    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy"
Nov 02 18:11:34.428709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428502    4456 cpu_manager.go:414] "containerMap: removing container" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server"
Nov 02 18:11:34.428709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428510    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server"
Nov 02 18:11:34.428709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428521    4456 cpu_manager.go:414] "containerMap: removing container" podUID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" containerName="payload-processing"
Nov 02 18:11:34.428709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428528    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" containerName="payload-processing"
Nov 02 18:11:34.429010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428556    4456 cpu_manager.go:414] "containerMap: removing container" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerName="payload-processing"
Nov 02 18:11:34.429010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428563    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerName="payload-processing"
Nov 02 18:11:34.429010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428577    4456 cpu_manager.go:414] "containerMap: removing container" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy"
Nov 02 18:11:34.429010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428584    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy"
Nov 02 18:11:34.429010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428599    4456 cpu_manager.go:414] "containerMap: removing container" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing"
Nov 02 18:11:34.429010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428606    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing"
Nov 02 18:11:34.429010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428617    4456 cpu_manager.go:414] "containerMap: removing container" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy"
Nov 02 18:11:34.429010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428624    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy"
Nov 02 18:11:34.429010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428637    4456 cpu_manager.go:414] "containerMap: removing container" podUID="22b556c1-112b-49d4-932c-0143e912a54e" containerName="payload-pre-processing"
Nov 02 18:11:34.429010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428644    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="22b556c1-112b-49d4-932c-0143e912a54e" containerName="payload-pre-processing"
Nov 02 18:11:34.429010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428656    4456 cpu_manager.go:414] "containerMap: removing container" podUID="44b66f41-99ae-446b-a50a-2ec0b71563a4" containerName="payload-pre-processing"
Nov 02 18:11:34.429287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428665    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="44b66f41-99ae-446b-a50a-2ec0b71563a4" containerName="payload-pre-processing"
Nov 02 18:11:34.429287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428679    4456 cpu_manager.go:414] "containerMap: removing container" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerName="payload-pre-processing"
Nov 02 18:11:34.429287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428686    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerName="payload-pre-processing"
Nov 02 18:11:34.429287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428718    4456 cpu_manager.go:414] "containerMap: removing container" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server"
Nov 02 18:11:34.429287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428725    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server"
Nov 02 18:11:34.429287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428742    4456 cpu_manager.go:414] "containerMap: removing container" podUID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" containerName="payload-processing"
Nov 02 18:11:34.429287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428749    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" containerName="payload-processing"
Nov 02 18:11:34.429287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428764    4456 cpu_manager.go:414] "containerMap: removing container" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerName="payload-pre-processing"
Nov 02 18:11:34.429287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428771    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerName="payload-pre-processing"
Nov 02 18:11:34.429287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428782    4456 cpu_manager.go:414] "containerMap: removing container" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerName="payload-processing"
Nov 02 18:11:34.429287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428789    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerName="payload-processing"
Nov 02 18:11:34.429553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428804    4456 cpu_manager.go:414] "containerMap: removing container" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing"
Nov 02 18:11:34.429553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.428832    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing"
Nov 02 18:11:34.429553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429067    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy"
Nov 02 18:11:34.429553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429097    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e141d845-db9d-4f94-b05e-cfb7848e8fef" containerName="payload-processing"
Nov 02 18:11:34.429553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429108    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="82ba53b4-e91a-4299-9498-f5abcd34576b" containerName="istio-proxy"
Nov 02 18:11:34.429553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429113    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="551602c4-254b-4532-8961-a98bf0f81089" containerName="istio-proxy"
Nov 02 18:11:34.429553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429127    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f4ee5c50-5738-496a-b5ef-c86ead76e0f6" containerName="payload-processing"
Nov 02 18:11:34.429553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429133    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="3c8d9150-66d1-4be8-8bca-3782fb0eac36" containerName="istio-proxy"
Nov 02 18:11:34.429553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429140    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy"
Nov 02 18:11:34.429553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429149    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d96e066-10c0-4fe2-9e24-ffd6195b22af" containerName="payload-pre-processing"
Nov 02 18:11:34.429553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429157    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server"
Nov 02 18:11:34.429553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429163    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy"
Nov 02 18:11:34.429887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429169    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="60c0e838-5c79-4bf7-a7c6-87903ae8bd8c" containerName="payload-processing"
Nov 02 18:11:34.429887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429175    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5ce3335c-36e6-4018-a264-5b79085f09f3" containerName="payload-processing"
Nov 02 18:11:34.429887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429182    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy"
Nov 02 18:11:34.429887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429189    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="96db7b02-b878-435a-869e-1d30e147c61d" containerName="payload-pre-processing"
Nov 02 18:11:34.429887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429195    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="32802945-5e87-4339-a7eb-33212aab7ba5" containerName="istio-proxy"
Nov 02 18:11:34.429887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429203    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="22b556c1-112b-49d4-932c-0143e912a54e" containerName="payload-pre-processing"
Nov 02 18:11:34.429887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429209    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="44b66f41-99ae-446b-a50a-2ec0b71563a4" containerName="payload-pre-processing"
Nov 02 18:11:34.429887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429216    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="3d398f17-735a-4d4b-b1cc-5b24fd266535" containerName="payload-pre-processing"
Nov 02 18:11:34.429887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429225    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6e78d6c7-c325-48f2-af90-804dade84d8f" containerName="download-server"
Nov 02 18:11:34.429887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429231    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="cf85b137-2b1e-4e22-8f6e-744a02ea4e46" containerName="payload-processing"
Nov 02 18:11:34.429887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429239    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e09749e-e648-40a3-a5f1-f0dc35fccf24" containerName="payload-pre-processing"
Nov 02 18:11:34.429887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429247    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="85814b5b-5d7d-4fe8-8c0e-a4f058f1f8d3" containerName="payload-processing"
Nov 02 18:11:34.430378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429253    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8c60dfaa-dabe-4726-91b5-165443ff3971" containerName="payload-processing"
Nov 02 18:11:34.430378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429263    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="0996b2fa-40e8-469c-9ace-75458691fa25" containerName="istio-proxy"
Nov 02 18:11:34.430378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429269    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="1c190514-ba33-4844-a9ec-e4bb47b5f706" containerName="payload-processing"
Nov 02 18:11:34.430378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.429276    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8064da0c-fc5d-493c-bcdf-4327716d4cc2" containerName="migrator"
Nov 02 18:11:34.432156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.432135    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"
Nov 02 18:11:34.450522 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.450494    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"]
Nov 02 18:11:34.456277 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.456252    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:11:34.456450 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.456430    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:11:34.553117 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.553084    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9\" (UID: \"8908e429-6925-4fc6-a3f0-6fbae9fa0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"
Nov 02 18:11:34.553362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.553344    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-58btd\" (UniqueName: \"kubernetes.io/projected/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-kube-api-access-58btd\") pod \"maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9\" (UID: \"8908e429-6925-4fc6-a3f0-6fbae9fa0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"
Nov 02 18:11:34.553496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.553483    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9\" (UID: \"8908e429-6925-4fc6-a3f0-6fbae9fa0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"
Nov 02 18:11:34.625177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.625140    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:34.654748 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.654717    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-58btd\" (UniqueName: \"kubernetes.io/projected/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-kube-api-access-58btd\") pod \"maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9\" (UID: \"8908e429-6925-4fc6-a3f0-6fbae9fa0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"
Nov 02 18:11:34.654952 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.654888    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9\" (UID: \"8908e429-6925-4fc6-a3f0-6fbae9fa0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"
Nov 02 18:11:34.655017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.654967    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9\" (UID: \"8908e429-6925-4fc6-a3f0-6fbae9fa0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"
Nov 02 18:11:34.655648 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.655620    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9\" (UID: \"8908e429-6925-4fc6-a3f0-6fbae9fa0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"
Nov 02 18:11:34.660751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.660725    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9\" (UID: \"8908e429-6925-4fc6-a3f0-6fbae9fa0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"
Nov 02 18:11:34.681547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.681512    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-58btd\" (UniqueName: \"kubernetes.io/projected/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-kube-api-access-58btd\") pod \"maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9\" (UID: \"8908e429-6925-4fc6-a3f0-6fbae9fa0d24\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"
Nov 02 18:11:34.752657 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.752587    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"
Nov 02 18:11:34.760457 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.760426    4456 generic.go:356] "Generic (PLEG): container finished" podID="0db36d92-4712-4338-958a-ebabf2c137ba" containerID="2501b584d5ee7323484f50ac5f32dd50c7fef23cf98bd9bf3120550c1d0985af" exitCode=0
Nov 02 18:11:34.760586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.760498    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" event={"ID":"0db36d92-4712-4338-958a-ebabf2c137ba","Type":"ContainerDied","Data":"2501b584d5ee7323484f50ac5f32dd50c7fef23cf98bd9bf3120550c1d0985af"}
Nov 02 18:11:34.760586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.760546    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-image-registry/image-registry-59767d8795-7vwkt" event={"ID":"0db36d92-4712-4338-958a-ebabf2c137ba","Type":"ContainerStarted","Data":"76aed64c2f8e1299a385d3394b57dac7a985f413e20ceba92b6e40c013918679"}
Nov 02 18:11:34.760671 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:34.760584    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 18:11:35.130095 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:35.130059    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8908e429_6925_4fc6_a3f0_6fbae9fa0d24.slice/crio-f7457f560e0cfec6b575a04ac9fb4176cd7cc1cf43629bfab3a764c34e56ae5f WatchSource:0}: Error finding container f7457f560e0cfec6b575a04ac9fb4176cd7cc1cf43629bfab3a764c34e56ae5f: Status 404 returned error can't find the container with id f7457f560e0cfec6b575a04ac9fb4176cd7cc1cf43629bfab3a764c34e56ae5f
Nov 02 18:11:35.130600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:35.130574    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:35.131589 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:35.131563    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"]
Nov 02 18:11:35.284308 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:35.284270    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:35.494712 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:35.494685    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 18:11:35.498991 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:35.498968    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-monitoring/metrics-server-7d987f75-ntp98"
Nov 02 18:11:35.770857 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:35.770806    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9" event={"ID":"8908e429-6925-4fc6-a3f0-6fbae9fa0d24","Type":"ContainerStarted","Data":"3950d86680a5cef109cf16b5f6c6460db4ed523ab353de3a891d0a896661c2ac"}
Nov 02 18:11:35.771018 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:35.770889    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9" event={"ID":"8908e429-6925-4fc6-a3f0-6fbae9fa0d24","Type":"ContainerStarted","Data":"f7457f560e0cfec6b575a04ac9fb4176cd7cc1cf43629bfab3a764c34e56ae5f"}
Nov 02 18:11:35.826440 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:35.826374    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9" podStartSLOduration=1.8183920759999999 podStartE2EDuration="1.818392076s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:35.813308313 +0000 UTC m=+2512.304371475" watchObservedRunningTime="2026-11-02 18:11:35.818392076 +0000 UTC m=+2512.309455280"
Nov 02 18:11:36.362648 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:36.362612    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:36.781554 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:36.781522    4456 generic.go:356] "Generic (PLEG): container finished" podID="8908e429-6925-4fc6-a3f0-6fbae9fa0d24" containerID="3950d86680a5cef109cf16b5f6c6460db4ed523ab353de3a891d0a896661c2ac" exitCode=0
Nov 02 18:11:36.781731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:36.781611    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9" event={"ID":"8908e429-6925-4fc6-a3f0-6fbae9fa0d24","Type":"ContainerDied","Data":"3950d86680a5cef109cf16b5f6c6460db4ed523ab353de3a891d0a896661c2ac"}
Nov 02 18:11:37.867318 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:37.867281    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:38.002362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.002329    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"
Nov 02 18:11:38.122898 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.122780    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-maas-api-openshift-ca\") pod \"8908e429-6925-4fc6-a3f0-6fbae9fa0d24\" (UID: \"8908e429-6925-4fc6-a3f0-6fbae9fa0d24\") "
Nov 02 18:11:38.122898 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.122853    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-kube-api-access-58btd\" (UniqueName: \"kubernetes.io/projected/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-kube-api-access-58btd\") pod \"8908e429-6925-4fc6-a3f0-6fbae9fa0d24\" (UID: \"8908e429-6925-4fc6-a3f0-6fbae9fa0d24\") "
Nov 02 18:11:38.123120 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.122952    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-maas-api-xks-ca\") pod \"8908e429-6925-4fc6-a3f0-6fbae9fa0d24\" (UID: \"8908e429-6925-4fc6-a3f0-6fbae9fa0d24\") "
Nov 02 18:11:38.123239 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.123211    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-maas-api-openshift-ca" pod "8908e429-6925-4fc6-a3f0-6fbae9fa0d24" (UID: "8908e429-6925-4fc6-a3f0-6fbae9fa0d24"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:38.123725 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.123706    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:38.126885 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.126852    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-kube-api-access-58btd" pod "8908e429-6925-4fc6-a3f0-6fbae9fa0d24" (UID: "8908e429-6925-4fc6-a3f0-6fbae9fa0d24"). InnerVolumeSpecName "kube-api-access-58btd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:38.126991 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.126883    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-maas-api-xks-ca" pod "8908e429-6925-4fc6-a3f0-6fbae9fa0d24" (UID: "8908e429-6925-4fc6-a3f0-6fbae9fa0d24"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:11:38.145504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.145435    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:38.227021 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.226458    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:38.227177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.227045    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-58btd\" (UniqueName: \"kubernetes.io/projected/8908e429-6925-4fc6-a3f0-6fbae9fa0d24-kube-api-access-58btd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:38.317118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.317083    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:38.747312 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.747275    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:38.818366 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.818332    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"
Nov 02 18:11:38.818638 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.818458    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9" event={"ID":"8908e429-6925-4fc6-a3f0-6fbae9fa0d24","Type":"ContainerDied","Data":"f7457f560e0cfec6b575a04ac9fb4176cd7cc1cf43629bfab3a764c34e56ae5f"}
Nov 02 18:11:38.818773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.818703    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f7457f560e0cfec6b575a04ac9fb4176cd7cc1cf43629bfab3a764c34e56ae5f"
Nov 02 18:11:38.898963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:38.898910    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:39.092435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.092327    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"]
Nov 02 18:11:39.105530 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.105491    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" containerID="cri-o://4a335e08a5bc2d765dbc826bbda03d9b544544393a5d585daa44175433b85831" gracePeriod=30
Nov 02 18:11:39.113320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.113294    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"]
Nov 02 18:11:39.113772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.113739    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy" containerID="cri-o://092d682533ef4dd4fce564dfead035598355d2c0e076d963e5b9852be346f785" gracePeriod=30
Nov 02 18:11:39.123684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.122836    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"]
Nov 02 18:11:39.123684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.123058    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy" containerID="cri-o://19ff9701b232887fc6c8cb51d7c49ea1498230cb1843842011b53e7a4b6f7336" gracePeriod=30
Nov 02 18:11:39.132597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.132574    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"]
Nov 02 18:11:39.132786 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.132765    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" containerID="cri-o://b40a2aae2b6ef5d220e6c8a56acd93bfa30d09d873cc836d366f512f7dc799f8" gracePeriod=30
Nov 02 18:11:39.146732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.146701    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:39.729339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.729305    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:39.745195 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.745163    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"]
Nov 02 18:11:39.745390 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.745359    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy" containerID="cri-o://321f32c6b20dd4c469c4740acf6ac58556d8ab5567e481fbe7905842a155c1f0" gracePeriod=30
Nov 02 18:11:39.779055 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.779027    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"]
Nov 02 18:11:39.779229 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.779207    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy" containerID="cri-o://33774b00458e580d97ee67142da150ad8d47b01d7485185ee8fef16f91745dda" gracePeriod=30
Nov 02 18:11:39.883958 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.883919    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"]
Nov 02 18:11:39.909919 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:39.884167    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy" containerID="cri-o://8fd11b5da302643890d03cf8b90f90991b0ace49793a57bde1469d40e28383be" gracePeriod=30
Nov 02 18:11:40.046709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.044365    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:40.077075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.068934    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"]
Nov 02 18:11:40.077075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.069167    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" containerID="cri-o://c492b110991299865769d96c519b16a0cdfd155200b1d4b35cdb7daa5de888f7" gracePeriod=30
Nov 02 18:11:40.077362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.077101    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"]
Nov 02 18:11:40.077362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.077318    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy" containerID="cri-o://8f711850377e64150821c64844dc9b4f619291b996bda5ba3ee277096962757c" gracePeriod=30
Nov 02 18:11:40.208840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.208363    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:40.447346 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.447318    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:40.727212 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.726857    4456 patch_prober.go:29] interesting pod/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:11:40.727212 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.726867    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 02 18:11:40.727212 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.726917    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:11:40.727212 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.726969    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:11:40.729910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.729884    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"]
Nov 02 18:11:40.730657 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.730638    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy"
Nov 02 18:11:40.730657 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.730658    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy"
Nov 02 18:11:40.730803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.730731    4456 cpu_manager.go:414] "containerMap: removing container" podUID="8908e429-6925-4fc6-a3f0-6fbae9fa0d24" containerName="revoke-keys"
Nov 02 18:11:40.730803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.730739    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8908e429-6925-4fc6-a3f0-6fbae9fa0d24" containerName="revoke-keys"
Nov 02 18:11:40.730976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.730960    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="67b961eb-ab19-4d8d-bb2a-db7c94a93436" containerName="istio-proxy"
Nov 02 18:11:40.731029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.730986    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4161075-440d-4373-b878-57c77612c6d2" containerName="istio-proxy"
Nov 02 18:11:40.731029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.731011    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8908e429-6925-4fc6-a3f0-6fbae9fa0d24" containerName="revoke-keys"
Nov 02 18:11:40.734326 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.734307    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"
Nov 02 18:11:40.741677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.741655    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:11:40.741767 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.741697    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:11:40.756205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.756184    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"]
Nov 02 18:11:40.776185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.775870    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d99092cc-b126-48db-8762-a4b8d64ad350-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-120394-29qk8\" (UID: \"d99092cc-b126-48db-8762-a4b8d64ad350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"
Nov 02 18:11:40.776331 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.776206    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-766cx\" (UniqueName: \"kubernetes.io/projected/d99092cc-b126-48db-8762-a4b8d64ad350-kube-api-access-766cx\") pod \"maas-api-revoke-keys-e2e-worker-w1-120394-29qk8\" (UID: \"d99092cc-b126-48db-8762-a4b8d64ad350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"
Nov 02 18:11:40.776331 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.776319    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d99092cc-b126-48db-8762-a4b8d64ad350-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-120394-29qk8\" (UID: \"d99092cc-b126-48db-8762-a4b8d64ad350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"
Nov 02 18:11:40.782480 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.782455    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:11:40.789404 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.789380    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:11:40.806067 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.805951    4456 patch_prober.go:29] interesting pod/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:11:40.806204 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.806098    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:11:40.873960 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.873923    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"]
Nov 02 18:11:40.877643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.877614    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d99092cc-b126-48db-8762-a4b8d64ad350-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-120394-29qk8\" (UID: \"d99092cc-b126-48db-8762-a4b8d64ad350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"
Nov 02 18:11:40.877887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.877845    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-766cx\" (UniqueName: \"kubernetes.io/projected/d99092cc-b126-48db-8762-a4b8d64ad350-kube-api-access-766cx\") pod \"maas-api-revoke-keys-e2e-worker-w1-120394-29qk8\" (UID: \"d99092cc-b126-48db-8762-a4b8d64ad350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"
Nov 02 18:11:40.877949 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.877910    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"
Nov 02 18:11:40.877949 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.877922    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d99092cc-b126-48db-8762-a4b8d64ad350-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-120394-29qk8\" (UID: \"d99092cc-b126-48db-8762-a4b8d64ad350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"
Nov 02 18:11:40.878283 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.878261    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d99092cc-b126-48db-8762-a4b8d64ad350-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-120394-29qk8\" (UID: \"d99092cc-b126-48db-8762-a4b8d64ad350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"
Nov 02 18:11:40.880851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.880830    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d99092cc-b126-48db-8762-a4b8d64ad350-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w1-120394-29qk8\" (UID: \"d99092cc-b126-48db-8762-a4b8d64ad350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"
Nov 02 18:11:40.904330 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.904294    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"]
Nov 02 18:11:40.910772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.910735    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-766cx\" (UniqueName: \"kubernetes.io/projected/d99092cc-b126-48db-8762-a4b8d64ad350-kube-api-access-766cx\") pod \"maas-api-revoke-keys-e2e-worker-w1-120394-29qk8\" (UID: \"d99092cc-b126-48db-8762-a4b8d64ad350\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"
Nov 02 18:11:40.979272 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.979199    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54bd962-e719-41c4-9c28-56460e46d0f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh\" (UID: \"f54bd962-e719-41c4-9c28-56460e46d0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"
Nov 02 18:11:40.979272 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.979245    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qxf8j\" (UniqueName: \"kubernetes.io/projected/f54bd962-e719-41c4-9c28-56460e46d0f6-kube-api-access-qxf8j\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh\" (UID: \"f54bd962-e719-41c4-9c28-56460e46d0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"
Nov 02 18:11:40.979478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:40.979430    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54bd962-e719-41c4-9c28-56460e46d0f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh\" (UID: \"f54bd962-e719-41c4-9c28-56460e46d0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"
Nov 02 18:11:41.050313 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:41.050272    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"
Nov 02 18:11:41.081362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:41.081331    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54bd962-e719-41c4-9c28-56460e46d0f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh\" (UID: \"f54bd962-e719-41c4-9c28-56460e46d0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"
Nov 02 18:11:41.081600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:41.081580    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54bd962-e719-41c4-9c28-56460e46d0f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh\" (UID: \"f54bd962-e719-41c4-9c28-56460e46d0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"
Nov 02 18:11:41.081670 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:41.081637    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qxf8j\" (UniqueName: \"kubernetes.io/projected/f54bd962-e719-41c4-9c28-56460e46d0f6-kube-api-access-qxf8j\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh\" (UID: \"f54bd962-e719-41c4-9c28-56460e46d0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"
Nov 02 18:11:41.082207 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:41.082178    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54bd962-e719-41c4-9c28-56460e46d0f6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh\" (UID: \"f54bd962-e719-41c4-9c28-56460e46d0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"
Nov 02 18:11:41.084605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:41.084578    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54bd962-e719-41c4-9c28-56460e46d0f6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh\" (UID: \"f54bd962-e719-41c4-9c28-56460e46d0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"
Nov 02 18:11:41.107789 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:41.107756    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qxf8j\" (UniqueName: \"kubernetes.io/projected/f54bd962-e719-41c4-9c28-56460e46d0f6-kube-api-access-qxf8j\") pod \"maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh\" (UID: \"f54bd962-e719-41c4-9c28-56460e46d0f6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"
Nov 02 18:11:41.243473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:41.243401    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"
Nov 02 18:11:41.376079 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:41.376049    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd99092cc_b126_48db_8762_a4b8d64ad350.slice/crio-cc707ac99c39d2ffde55c25963d1d558852b4e760d31b7fdabe5d54b375c4bc5 WatchSource:0}: Error finding container cc707ac99c39d2ffde55c25963d1d558852b4e760d31b7fdabe5d54b375c4bc5: Status 404 returned error can't find the container with id cc707ac99c39d2ffde55c25963d1d558852b4e760d31b7fdabe5d54b375c4bc5
Nov 02 18:11:41.376176 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:41.376139    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"]
Nov 02 18:11:41.579663 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:41.579628    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf54bd962_e719_41c4_9c28_56460e46d0f6.slice/crio-5c6877949a7de326d4e6b661788980388fc687c640563d9099516e68d53efb4e WatchSource:0}: Error finding container 5c6877949a7de326d4e6b661788980388fc687c640563d9099516e68d53efb4e: Status 404 returned error can't find the container with id 5c6877949a7de326d4e6b661788980388fc687c640563d9099516e68d53efb4e
Nov 02 18:11:41.581062 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:41.581036    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"]
Nov 02 18:11:41.975002 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:41.974965    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8" event={"ID":"d99092cc-b126-48db-8762-a4b8d64ad350","Type":"ContainerStarted","Data":"0b5be95732ef78cdc1af5066503c626a4c9cb8499de4dbb50308f359c6358fc4"}
Nov 02 18:11:41.975470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:41.975033    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8" event={"ID":"d99092cc-b126-48db-8762-a4b8d64ad350","Type":"ContainerStarted","Data":"cc707ac99c39d2ffde55c25963d1d558852b4e760d31b7fdabe5d54b375c4bc5"}
Nov 02 18:11:41.976486 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:41.976455    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh" event={"ID":"f54bd962-e719-41c4-9c28-56460e46d0f6","Type":"ContainerStarted","Data":"5c6877949a7de326d4e6b661788980388fc687c640563d9099516e68d53efb4e"}
Nov 02 18:11:42.021394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:42.021342    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8" podStartSLOduration=2.021328526 podStartE2EDuration="2.021328526s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:42.017831464 +0000 UTC m=+2518.508894622" watchObservedRunningTime="2026-11-02 18:11:42.021328526 +0000 UTC m=+2518.512391734"
Nov 02 18:11:42.153045 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:42.152958    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:42.324095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:42.324049    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:42.991075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:42.991037    4456 generic.go:356] "Generic (PLEG): container finished" podID="d99092cc-b126-48db-8762-a4b8d64ad350" containerID="0b5be95732ef78cdc1af5066503c626a4c9cb8499de4dbb50308f359c6358fc4" exitCode=0
Nov 02 18:11:42.991545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:42.991232    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8" event={"ID":"d99092cc-b126-48db-8762-a4b8d64ad350","Type":"ContainerDied","Data":"0b5be95732ef78cdc1af5066503c626a4c9cb8499de4dbb50308f359c6358fc4"}
Nov 02 18:11:42.993476 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:42.993447    4456 generic.go:356] "Generic (PLEG): container finished" podID="f54bd962-e719-41c4-9c28-56460e46d0f6" containerID="c62f25f4e3ed9c7ecded4a01aec626d0b95966058426c9da278a522fb99b9280" exitCode=0
Nov 02 18:11:42.993584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:42.993490    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh" event={"ID":"f54bd962-e719-41c4-9c28-56460e46d0f6","Type":"ContainerDied","Data":"c62f25f4e3ed9c7ecded4a01aec626d0b95966058426c9da278a522fb99b9280"}
Nov 02 18:11:44.047139 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.047100    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:44.477442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.477288    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"
Nov 02 18:11:44.495972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.489318    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"
Nov 02 18:11:44.495972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.493372    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:11:44.542150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.541988    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f54bd962-e719-41c4-9c28-56460e46d0f6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54bd962-e719-41c4-9c28-56460e46d0f6-maas-api-xks-ca\") pod \"f54bd962-e719-41c4-9c28-56460e46d0f6\" (UID: \"f54bd962-e719-41c4-9c28-56460e46d0f6\") "
Nov 02 18:11:44.542150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.542072    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-istio-envoy\") pod \"ef6443b3-b452-4868-9cde-f762cf522284\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") "
Nov 02 18:11:44.542662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.542555    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-credential-socket\") pod \"ef6443b3-b452-4868-9cde-f762cf522284\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") "
Nov 02 18:11:44.542662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.542620    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d99092cc-b126-48db-8762-a4b8d64ad350-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d99092cc-b126-48db-8762-a4b8d64ad350-maas-api-xks-ca\") pod \"d99092cc-b126-48db-8762-a4b8d64ad350\" (UID: \"d99092cc-b126-48db-8762-a4b8d64ad350\") "
Nov 02 18:11:44.542996 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.542957    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f54bd962-e719-41c4-9c28-56460e46d0f6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54bd962-e719-41c4-9c28-56460e46d0f6-maas-api-openshift-ca\") pod \"f54bd962-e719-41c4-9c28-56460e46d0f6\" (UID: \"f54bd962-e719-41c4-9c28-56460e46d0f6\") "
Nov 02 18:11:44.543194 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.543156    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ef6443b3-b452-4868-9cde-f762cf522284-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6443b3-b452-4868-9cde-f762cf522284-istio-podinfo\") pod \"ef6443b3-b452-4868-9cde-f762cf522284\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") "
Nov 02 18:11:44.543416 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.543327    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-workload-socket\") pod \"ef6443b3-b452-4868-9cde-f762cf522284\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") "
Nov 02 18:11:44.543801 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.543540    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d99092cc-b126-48db-8762-a4b8d64ad350-kube-api-access-766cx\" (UniqueName: \"kubernetes.io/projected/d99092cc-b126-48db-8762-a4b8d64ad350-kube-api-access-766cx\") pod \"d99092cc-b126-48db-8762-a4b8d64ad350\" (UID: \"d99092cc-b126-48db-8762-a4b8d64ad350\") "
Nov 02 18:11:44.543801 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.543678    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6443b3-b452-4868-9cde-f762cf522284-kube-api-access-rhfkr\" (UniqueName: \"kubernetes.io/projected/ef6443b3-b452-4868-9cde-f762cf522284-kube-api-access-rhfkr\") pod \"ef6443b3-b452-4868-9cde-f762cf522284\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") "
Nov 02 18:11:44.543801 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.543758    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ef6443b3-b452-4868-9cde-f762cf522284-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6443b3-b452-4868-9cde-f762cf522284-istiod-ca-cert\") pod \"ef6443b3-b452-4868-9cde-f762cf522284\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") "
Nov 02 18:11:44.545324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.544100    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-istio-data\") pod \"ef6443b3-b452-4868-9cde-f762cf522284\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") "
Nov 02 18:11:44.545324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.544160    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-workload-certs\") pod \"ef6443b3-b452-4868-9cde-f762cf522284\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") "
Nov 02 18:11:44.545324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.544193    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d99092cc-b126-48db-8762-a4b8d64ad350-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d99092cc-b126-48db-8762-a4b8d64ad350-maas-api-openshift-ca\") pod \"d99092cc-b126-48db-8762-a4b8d64ad350\" (UID: \"d99092cc-b126-48db-8762-a4b8d64ad350\") "
Nov 02 18:11:44.545324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.544315    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f54bd962-e719-41c4-9c28-56460e46d0f6-kube-api-access-qxf8j\" (UniqueName: \"kubernetes.io/projected/f54bd962-e719-41c4-9c28-56460e46d0f6-kube-api-access-qxf8j\") pod \"f54bd962-e719-41c4-9c28-56460e46d0f6\" (UID: \"f54bd962-e719-41c4-9c28-56460e46d0f6\") "
Nov 02 18:11:44.545324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.544421    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ef6443b3-b452-4868-9cde-f762cf522284-istio-token\" (UniqueName: \"kubernetes.io/projected/ef6443b3-b452-4868-9cde-f762cf522284-istio-token\") pod \"ef6443b3-b452-4868-9cde-f762cf522284\" (UID: \"ef6443b3-b452-4868-9cde-f762cf522284\") "
Nov 02 18:11:44.547087 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.546756    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-workload-socket" pod "ef6443b3-b452-4868-9cde-f762cf522284" (UID: "ef6443b3-b452-4868-9cde-f762cf522284"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.551149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.548257    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f54bd962-e719-41c4-9c28-56460e46d0f6-maas-api-openshift-ca" pod "f54bd962-e719-41c4-9c28-56460e46d0f6" (UID: "f54bd962-e719-41c4-9c28-56460e46d0f6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:44.551149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.548539    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-workload-certs" pod "ef6443b3-b452-4868-9cde-f762cf522284" (UID: "ef6443b3-b452-4868-9cde-f762cf522284"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.551149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.549646    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f54bd962-e719-41c4-9c28-56460e46d0f6-maas-api-xks-ca" pod "f54bd962-e719-41c4-9c28-56460e46d0f6" (UID: "f54bd962-e719-41c4-9c28-56460e46d0f6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:11:44.551149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.549739    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-credential-socket" pod "ef6443b3-b452-4868-9cde-f762cf522284" (UID: "ef6443b3-b452-4868-9cde-f762cf522284"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.551149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.550330    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-istio-data" pod "ef6443b3-b452-4868-9cde-f762cf522284" (UID: "ef6443b3-b452-4868-9cde-f762cf522284"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.551464 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.550796    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ef6443b3-b452-4868-9cde-f762cf522284-istiod-ca-cert" pod "ef6443b3-b452-4868-9cde-f762cf522284" (UID: "ef6443b3-b452-4868-9cde-f762cf522284"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:44.551942 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.551915    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d99092cc-b126-48db-8762-a4b8d64ad350-maas-api-openshift-ca" pod "d99092cc-b126-48db-8762-a4b8d64ad350" (UID: "d99092cc-b126-48db-8762-a4b8d64ad350"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:44.551942 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.551929    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-istio-envoy" pod "ef6443b3-b452-4868-9cde-f762cf522284" (UID: "ef6443b3-b452-4868-9cde-f762cf522284"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.554133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.554108    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ef6443b3-b452-4868-9cde-f762cf522284-istio-podinfo" pod "ef6443b3-b452-4868-9cde-f762cf522284" (UID: "ef6443b3-b452-4868-9cde-f762cf522284"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:44.555488 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.554340    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6443b3-b452-4868-9cde-f762cf522284-kube-api-access-rhfkr" pod "ef6443b3-b452-4868-9cde-f762cf522284" (UID: "ef6443b3-b452-4868-9cde-f762cf522284"). InnerVolumeSpecName "kube-api-access-rhfkr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:44.555488 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.554786    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d99092cc-b126-48db-8762-a4b8d64ad350-maas-api-xks-ca" pod "d99092cc-b126-48db-8762-a4b8d64ad350" (UID: "d99092cc-b126-48db-8762-a4b8d64ad350"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:11:44.563483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.563205    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f54bd962-e719-41c4-9c28-56460e46d0f6-kube-api-access-qxf8j" pod "f54bd962-e719-41c4-9c28-56460e46d0f6" (UID: "f54bd962-e719-41c4-9c28-56460e46d0f6"). InnerVolumeSpecName "kube-api-access-qxf8j". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:44.563945 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.563919    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d99092cc-b126-48db-8762-a4b8d64ad350-kube-api-access-766cx" pod "d99092cc-b126-48db-8762-a4b8d64ad350" (UID: "d99092cc-b126-48db-8762-a4b8d64ad350"). InnerVolumeSpecName "kube-api-access-766cx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:44.567326 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.567301    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ef6443b3-b452-4868-9cde-f762cf522284-istio-token" pod "ef6443b3-b452-4868-9cde-f762cf522284" (UID: "ef6443b3-b452-4868-9cde-f762cf522284"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:44.622352 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.622317    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 18:11:44.639487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.639458    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 18:11:44.647912 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.647882    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-workload-socket\") pod \"778b9a36-6a7b-4251-a91a-999e358d8c5c\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") "
Nov 02 18:11:44.648021 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.647988    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-token\" (UniqueName: \"kubernetes.io/projected/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-token\") pod \"778b9a36-6a7b-4251-a91a-999e358d8c5c\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") "
Nov 02 18:11:44.648109 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.648091    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/778b9a36-6a7b-4251-a91a-999e358d8c5c-kube-api-access-wz4ld\" (UniqueName: \"kubernetes.io/projected/778b9a36-6a7b-4251-a91a-999e358d8c5c-kube-api-access-wz4ld\") pod \"778b9a36-6a7b-4251-a91a-999e358d8c5c\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") "
Nov 02 18:11:44.648170 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.648128    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-data\") pod \"778b9a36-6a7b-4251-a91a-999e358d8c5c\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") "
Nov 02 18:11:44.648225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.648184    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-podinfo\") pod \"778b9a36-6a7b-4251-a91a-999e358d8c5c\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") "
Nov 02 18:11:44.648225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.648213    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/778b9a36-6a7b-4251-a91a-999e358d8c5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778b9a36-6a7b-4251-a91a-999e358d8c5c-istiod-ca-cert\") pod \"778b9a36-6a7b-4251-a91a-999e358d8c5c\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") "
Nov 02 18:11:44.648318 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.648241    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-credential-socket\") pod \"778b9a36-6a7b-4251-a91a-999e358d8c5c\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") "
Nov 02 18:11:44.648427 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.648334    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-workload-certs\") pod \"778b9a36-6a7b-4251-a91a-999e358d8c5c\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") "
Nov 02 18:11:44.648529 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.648510    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-envoy\") pod \"778b9a36-6a7b-4251-a91a-999e358d8c5c\" (UID: \"778b9a36-6a7b-4251-a91a-999e358d8c5c\") "
Nov 02 18:11:44.649182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.649161    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qxf8j\" (UniqueName: \"kubernetes.io/projected/f54bd962-e719-41c4-9c28-56460e46d0f6-kube-api-access-qxf8j\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.649182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.649183    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ef6443b3-b452-4868-9cde-f762cf522284-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.649335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.649192    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/f54bd962-e719-41c4-9c28-56460e46d0f6-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.649335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.649200    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.649335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.649208    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.649335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.649216    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d99092cc-b126-48db-8762-a4b8d64ad350-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.649335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.649267    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/f54bd962-e719-41c4-9c28-56460e46d0f6-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.649335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.649286    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ef6443b3-b452-4868-9cde-f762cf522284-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.649335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.649295    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.649674 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.649303    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-766cx\" (UniqueName: \"kubernetes.io/projected/d99092cc-b126-48db-8762-a4b8d64ad350-kube-api-access-766cx\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.649674 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.649312    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rhfkr\" (UniqueName: \"kubernetes.io/projected/ef6443b3-b452-4868-9cde-f762cf522284-kube-api-access-rhfkr\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.649674 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.649320    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ef6443b3-b452-4868-9cde-f762cf522284-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.649674 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.649328    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.649674 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.649345    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ef6443b3-b452-4868-9cde-f762cf522284-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.649674 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.649353    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d99092cc-b126-48db-8762-a4b8d64ad350-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.651489 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.651245    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-data" pod "778b9a36-6a7b-4251-a91a-999e358d8c5c" (UID: "778b9a36-6a7b-4251-a91a-999e358d8c5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.651489 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.651437    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/778b9a36-6a7b-4251-a91a-999e358d8c5c-istiod-ca-cert" pod "778b9a36-6a7b-4251-a91a-999e358d8c5c" (UID: "778b9a36-6a7b-4251-a91a-999e358d8c5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:44.651730 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.651599    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-workload-socket" pod "778b9a36-6a7b-4251-a91a-999e358d8c5c" (UID: "778b9a36-6a7b-4251-a91a-999e358d8c5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.651790 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.651741    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-credential-socket" pod "778b9a36-6a7b-4251-a91a-999e358d8c5c" (UID: "778b9a36-6a7b-4251-a91a-999e358d8c5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.652068 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.652044    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-workload-certs" pod "778b9a36-6a7b-4251-a91a-999e358d8c5c" (UID: "778b9a36-6a7b-4251-a91a-999e358d8c5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.657520 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.657491    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 18:11:44.658093 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.658057    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-podinfo" pod "778b9a36-6a7b-4251-a91a-999e358d8c5c" (UID: "778b9a36-6a7b-4251-a91a-999e358d8c5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:44.660456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.660119    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-envoy" pod "778b9a36-6a7b-4251-a91a-999e358d8c5c" (UID: "778b9a36-6a7b-4251-a91a-999e358d8c5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.660456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.660156    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778b9a36-6a7b-4251-a91a-999e358d8c5c-kube-api-access-wz4ld" pod "778b9a36-6a7b-4251-a91a-999e358d8c5c" (UID: "778b9a36-6a7b-4251-a91a-999e358d8c5c"). InnerVolumeSpecName "kube-api-access-wz4ld". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:44.661569 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.661545    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-token" pod "778b9a36-6a7b-4251-a91a-999e358d8c5c" (UID: "778b9a36-6a7b-4251-a91a-999e358d8c5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:44.750185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750153    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-kube-api-access-rstmc\" (UniqueName: \"kubernetes.io/projected/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-kube-api-access-rstmc\") pod \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") "
Nov 02 18:11:44.750409 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750201    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-workload-certs\") pod \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") "
Nov 02 18:11:44.750409 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750236    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-credential-socket\") pod \"408afe2d-75a0-4537-b58c-9ab31f161685\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") "
Nov 02 18:11:44.750409 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750265    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-workload-certs\") pod \"408afe2d-75a0-4537-b58c-9ab31f161685\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") "
Nov 02 18:11:44.750409 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750313    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408afe2d-75a0-4537-b58c-9ab31f161685-istio-token\" (UniqueName: \"kubernetes.io/projected/408afe2d-75a0-4537-b58c-9ab31f161685-istio-token\") pod \"408afe2d-75a0-4537-b58c-9ab31f161685\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") "
Nov 02 18:11:44.750409 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750344    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istiod-ca-cert\") pod \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") "
Nov 02 18:11:44.750650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750371    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/408afe2d-75a0-4537-b58c-9ab31f161685-kube-api-access-hm9fl\" (UniqueName: \"kubernetes.io/projected/408afe2d-75a0-4537-b58c-9ab31f161685-kube-api-access-hm9fl\") pod \"408afe2d-75a0-4537-b58c-9ab31f161685\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") "
Nov 02 18:11:44.750650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750416    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-istio-envoy\") pod \"408afe2d-75a0-4537-b58c-9ab31f161685\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") "
Nov 02 18:11:44.750650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750447    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-envoy\") pod \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") "
Nov 02 18:11:44.750650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750579    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-data\") pod \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") "
Nov 02 18:11:44.750650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750609    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-istio-data\") pod \"408afe2d-75a0-4537-b58c-9ab31f161685\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") "
Nov 02 18:11:44.750876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750657    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-credential-socket\") pod \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") "
Nov 02 18:11:44.750876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750682    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/408afe2d-75a0-4537-b58c-9ab31f161685-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408afe2d-75a0-4537-b58c-9ab31f161685-istio-podinfo\") pod \"408afe2d-75a0-4537-b58c-9ab31f161685\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") "
Nov 02 18:11:44.750876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750721    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-workload-socket\") pod \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") "
Nov 02 18:11:44.750876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750760    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-podinfo\") pod \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") "
Nov 02 18:11:44.750876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750850    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-workload-socket\") pod \"408afe2d-75a0-4537-b58c-9ab31f161685\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") "
Nov 02 18:11:44.751179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750869    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-token\" (UniqueName: \"kubernetes.io/projected/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-token\") pod \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\" (UID: \"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21\") "
Nov 02 18:11:44.751179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750871    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-workload-certs" pod "408afe2d-75a0-4537-b58c-9ab31f161685" (UID: "408afe2d-75a0-4537-b58c-9ab31f161685"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.751179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.750884    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/408afe2d-75a0-4537-b58c-9ab31f161685-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408afe2d-75a0-4537-b58c-9ab31f161685-istiod-ca-cert\") pod \"408afe2d-75a0-4537-b58c-9ab31f161685\" (UID: \"408afe2d-75a0-4537-b58c-9ab31f161685\") "
Nov 02 18:11:44.751290 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.751209    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/408afe2d-75a0-4537-b58c-9ab31f161685-istiod-ca-cert" pod "408afe2d-75a0-4537-b58c-9ab31f161685" (UID: "408afe2d-75a0-4537-b58c-9ab31f161685"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:44.752286 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.752257    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/408afe2d-75a0-4537-b58c-9ab31f161685-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.752286 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.752292    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.752526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.752307    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.752526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.752320    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.752526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.752379    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.752526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.752393    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wz4ld\" (UniqueName: \"kubernetes.io/projected/778b9a36-6a7b-4251-a91a-999e358d8c5c-kube-api-access-wz4ld\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.752526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.752410    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.752526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.752424    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/778b9a36-6a7b-4251-a91a-999e358d8c5c-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.752526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.752437    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/778b9a36-6a7b-4251-a91a-999e358d8c5c-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.752526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.752449    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.752981 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.752461    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/778b9a36-6a7b-4251-a91a-999e358d8c5c-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.752981 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.752723    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istiod-ca-cert" pod "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" (UID: "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:44.753905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.753878    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408afe2d-75a0-4537-b58c-9ab31f161685-istio-token" pod "408afe2d-75a0-4537-b58c-9ab31f161685" (UID: "408afe2d-75a0-4537-b58c-9ab31f161685"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:44.754146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.754123    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-workload-certs" pod "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" (UID: "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.754376 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.754353    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-credential-socket" pod "408afe2d-75a0-4537-b58c-9ab31f161685" (UID: "408afe2d-75a0-4537-b58c-9ab31f161685"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.754586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.754562    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-credential-socket" pod "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" (UID: "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.760962 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.760334    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-workload-socket" pod "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" (UID: "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.760962 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.760614    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-data" pod "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" (UID: "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.761711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.761679    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/408afe2d-75a0-4537-b58c-9ab31f161685-istio-podinfo" pod "408afe2d-75a0-4537-b58c-9ab31f161685" (UID: "408afe2d-75a0-4537-b58c-9ab31f161685"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:44.761790 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.761739    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-podinfo" pod "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" (UID: "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:44.768178 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.768144    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-istio-data" pod "408afe2d-75a0-4537-b58c-9ab31f161685" (UID: "408afe2d-75a0-4537-b58c-9ab31f161685"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.768178 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.768160    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-workload-socket" pod "408afe2d-75a0-4537-b58c-9ab31f161685" (UID: "408afe2d-75a0-4537-b58c-9ab31f161685"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.781465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.781411    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-istio-envoy" pod "408afe2d-75a0-4537-b58c-9ab31f161685" (UID: "408afe2d-75a0-4537-b58c-9ab31f161685"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.781612 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.781510    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/408afe2d-75a0-4537-b58c-9ab31f161685-kube-api-access-hm9fl" pod "408afe2d-75a0-4537-b58c-9ab31f161685" (UID: "408afe2d-75a0-4537-b58c-9ab31f161685"). InnerVolumeSpecName "kube-api-access-hm9fl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:44.781959 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.781922    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-kube-api-access-rstmc" pod "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" (UID: "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21"). InnerVolumeSpecName "kube-api-access-rstmc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:44.785185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.785003    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-envoy" pod "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" (UID: "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:44.792314 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.792232    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-token" pod "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" (UID: "6aa41ecd-4a93-4b17-8e84-3f3c018c4d21"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:44.854050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.853945    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.854050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.853977    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.854050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.853991    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.854050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.854005    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/408afe2d-75a0-4537-b58c-9ab31f161685-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.854050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.854018    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.854050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.854030    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.854050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.854042    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.854050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.854054    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.854664 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.854067    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rstmc\" (UniqueName: \"kubernetes.io/projected/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-kube-api-access-rstmc\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.854664 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.854080    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.854664 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.854094    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.854664 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.854106    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/408afe2d-75a0-4537-b58c-9ab31f161685-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.854664 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.854119    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.854664 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.854131    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hm9fl\" (UniqueName: \"kubernetes.io/projected/408afe2d-75a0-4537-b58c-9ab31f161685-kube-api-access-hm9fl\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.854664 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.854143    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/408afe2d-75a0-4537-b58c-9ab31f161685-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.854664 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.854157    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:44.974150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:44.974117    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"]
Nov 02 18:11:45.051367 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.050728    4456 generic.go:356] "Generic (PLEG): container finished" podID="ef6443b3-b452-4868-9cde-f762cf522284" containerID="4a335e08a5bc2d765dbc826bbda03d9b544544393a5d585daa44175433b85831" exitCode=0
Nov 02 18:11:45.051367 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.050890    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" event={"ID":"ef6443b3-b452-4868-9cde-f762cf522284","Type":"ContainerDied","Data":"4a335e08a5bc2d765dbc826bbda03d9b544544393a5d585daa44175433b85831"}
Nov 02 18:11:45.051367 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.050939    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh" event={"ID":"ef6443b3-b452-4868-9cde-f762cf522284","Type":"ContainerDied","Data":"6cd905c99e4662f4685dd3c073d95e4f2542965be0683c159b28a2a71070a147"}
Nov 02 18:11:45.051367 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.050968    4456 scope.go:122] "RemoveContainer" containerID="4a335e08a5bc2d765dbc826bbda03d9b544544393a5d585daa44175433b85831"
Nov 02 18:11:45.051367 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.051140    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"
Nov 02 18:11:45.060951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.060903    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh" event={"ID":"f54bd962-e719-41c4-9c28-56460e46d0f6","Type":"ContainerDied","Data":"5c6877949a7de326d4e6b661788980388fc687c640563d9099516e68d53efb4e"}
Nov 02 18:11:45.061056 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.060965    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5c6877949a7de326d4e6b661788980388fc687c640563d9099516e68d53efb4e"
Nov 02 18:11:45.061056 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.061038    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"
Nov 02 18:11:45.074273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.073155    4456 generic.go:356] "Generic (PLEG): container finished" podID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerID="321f32c6b20dd4c469c4740acf6ac58556d8ab5567e481fbe7905842a155c1f0" exitCode=0
Nov 02 18:11:45.074273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.073247    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" event={"ID":"73849b3d-673e-41a8-aff1-db9f0a9cce85","Type":"ContainerDied","Data":"321f32c6b20dd4c469c4740acf6ac58556d8ab5567e481fbe7905842a155c1f0"}
Nov 02 18:11:45.082009 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.081951    4456 generic.go:356] "Generic (PLEG): container finished" podID="408afe2d-75a0-4537-b58c-9ab31f161685" containerID="19ff9701b232887fc6c8cb51d7c49ea1498230cb1843842011b53e7a4b6f7336" exitCode=0
Nov 02 18:11:45.084098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.082309    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" event={"ID":"408afe2d-75a0-4537-b58c-9ab31f161685","Type":"ContainerDied","Data":"19ff9701b232887fc6c8cb51d7c49ea1498230cb1843842011b53e7a4b6f7336"}
Nov 02 18:11:45.084098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.082360    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz" event={"ID":"408afe2d-75a0-4537-b58c-9ab31f161685","Type":"ContainerDied","Data":"cb1df1cbaf8bfb6078cbda186b01ed7e40095d1b0ba50d11718114d10c543589"}
Nov 02 18:11:45.084098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.082446    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"
Nov 02 18:11:45.088095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.087341    4456 generic.go:356] "Generic (PLEG): container finished" podID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerID="092d682533ef4dd4fce564dfead035598355d2c0e076d963e5b9852be346f785" exitCode=0
Nov 02 18:11:45.088095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.087441    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" event={"ID":"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21","Type":"ContainerDied","Data":"092d682533ef4dd4fce564dfead035598355d2c0e076d963e5b9852be346f785"}
Nov 02 18:11:45.088095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.087480    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8" event={"ID":"6aa41ecd-4a93-4b17-8e84-3f3c018c4d21","Type":"ContainerDied","Data":"e781bf2d2bc7c1a28d8f91cdbfb3c720bee5b614f7581891fa084fcebe1a2d6b"}
Nov 02 18:11:45.088095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.087560    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"
Nov 02 18:11:45.125237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.116416    4456 generic.go:356] "Generic (PLEG): container finished" podID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerID="b40a2aae2b6ef5d220e6c8a56acd93bfa30d09d873cc836d366f512f7dc799f8" exitCode=0
Nov 02 18:11:45.125237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.116643    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"
Nov 02 18:11:45.125237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.117390    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" event={"ID":"778b9a36-6a7b-4251-a91a-999e358d8c5c","Type":"ContainerDied","Data":"b40a2aae2b6ef5d220e6c8a56acd93bfa30d09d873cc836d366f512f7dc799f8"}
Nov 02 18:11:45.125237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.117445    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq" event={"ID":"778b9a36-6a7b-4251-a91a-999e358d8c5c","Type":"ContainerDied","Data":"d903a26c6171a2534f2d5af1688376f441f68b47cb471efc27be8ad241043786"}
Nov 02 18:11:45.125237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.121204    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8" event={"ID":"d99092cc-b126-48db-8762-a4b8d64ad350","Type":"ContainerDied","Data":"cc707ac99c39d2ffde55c25963d1d558852b4e760d31b7fdabe5d54b375c4bc5"}
Nov 02 18:11:45.125237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.121245    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cc707ac99c39d2ffde55c25963d1d558852b4e760d31b7fdabe5d54b375c4bc5"
Nov 02 18:11:45.125237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.121320    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"
Nov 02 18:11:45.136775 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.136741    4456 generic.go:356] "Generic (PLEG): container finished" podID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerID="33774b00458e580d97ee67142da150ad8d47b01d7485185ee8fef16f91745dda" exitCode=0
Nov 02 18:11:45.137576 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.136874    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" event={"ID":"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d","Type":"ContainerDied","Data":"33774b00458e580d97ee67142da150ad8d47b01d7485185ee8fef16f91745dda"}
Nov 02 18:11:45.140317 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.140289    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:45.306015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.303889    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:45.361840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.361218    4456 scope.go:122] "RemoveContainer" containerID="4a335e08a5bc2d765dbc826bbda03d9b544544393a5d585daa44175433b85831"
Nov 02 18:11:45.369043 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.362973    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 18:11:45.377694 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:45.377482    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a335e08a5bc2d765dbc826bbda03d9b544544393a5d585daa44175433b85831\": container with ID starting with 4a335e08a5bc2d765dbc826bbda03d9b544544393a5d585daa44175433b85831 not found: ID does not exist" containerID="4a335e08a5bc2d765dbc826bbda03d9b544544393a5d585daa44175433b85831"
Nov 02 18:11:45.377694 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.377530    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a335e08a5bc2d765dbc826bbda03d9b544544393a5d585daa44175433b85831"} err="failed to get container status \"4a335e08a5bc2d765dbc826bbda03d9b544544393a5d585daa44175433b85831\": rpc error: code = NotFound desc = could not find container \"4a335e08a5bc2d765dbc826bbda03d9b544544393a5d585daa44175433b85831\": container with ID starting with 4a335e08a5bc2d765dbc826bbda03d9b544544393a5d585daa44175433b85831 not found: ID does not exist"
Nov 02 18:11:45.377694 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.377557    4456 scope.go:122] "RemoveContainer" containerID="19ff9701b232887fc6c8cb51d7c49ea1498230cb1843842011b53e7a4b6f7336"
Nov 02 18:11:45.378184 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.378130    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"]
Nov 02 18:11:45.386980 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.386917    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-jkcgh"]
Nov 02 18:11:45.398499 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.398374    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 18:11:45.402149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.401513    4456 scope.go:122] "RemoveContainer" containerID="19ff9701b232887fc6c8cb51d7c49ea1498230cb1843842011b53e7a4b6f7336"
Nov 02 18:11:45.402149 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:45.401958    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"19ff9701b232887fc6c8cb51d7c49ea1498230cb1843842011b53e7a4b6f7336\": container with ID starting with 19ff9701b232887fc6c8cb51d7c49ea1498230cb1843842011b53e7a4b6f7336 not found: ID does not exist" containerID="19ff9701b232887fc6c8cb51d7c49ea1498230cb1843842011b53e7a4b6f7336"
Nov 02 18:11:45.402149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.401994    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"19ff9701b232887fc6c8cb51d7c49ea1498230cb1843842011b53e7a4b6f7336"} err="failed to get container status \"19ff9701b232887fc6c8cb51d7c49ea1498230cb1843842011b53e7a4b6f7336\": rpc error: code = NotFound desc = could not find container \"19ff9701b232887fc6c8cb51d7c49ea1498230cb1843842011b53e7a4b6f7336\": container with ID starting with 19ff9701b232887fc6c8cb51d7c49ea1498230cb1843842011b53e7a4b6f7336 not found: ID does not exist"
Nov 02 18:11:45.402149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.402021    4456 scope.go:122] "RemoveContainer" containerID="092d682533ef4dd4fce564dfead035598355d2c0e076d963e5b9852be346f785"
Nov 02 18:11:45.420267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.420236    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"]
Nov 02 18:11:45.432147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.432119    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"]
Nov 02 18:11:45.438624 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.438596    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-l6tqz"]
Nov 02 18:11:45.438778 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.438749    4456 scope.go:122] "RemoveContainer" containerID="026fbcc527da33cce56513069ee46f158e36543f79011d0d0036bc53fe44819c"
Nov 02 18:11:45.452069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.451619    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"]
Nov 02 18:11:45.458802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.458468    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"]
Nov 02 18:11:45.461218 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.460941    4456 scope.go:122] "RemoveContainer" containerID="092d682533ef4dd4fce564dfead035598355d2c0e076d963e5b9852be346f785"
Nov 02 18:11:45.461729 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:45.461463    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"092d682533ef4dd4fce564dfead035598355d2c0e076d963e5b9852be346f785\": container with ID starting with 092d682533ef4dd4fce564dfead035598355d2c0e076d963e5b9852be346f785 not found: ID does not exist" containerID="092d682533ef4dd4fce564dfead035598355d2c0e076d963e5b9852be346f785"
Nov 02 18:11:45.461729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.461501    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"092d682533ef4dd4fce564dfead035598355d2c0e076d963e5b9852be346f785"} err="failed to get container status \"092d682533ef4dd4fce564dfead035598355d2c0e076d963e5b9852be346f785\": rpc error: code = NotFound desc = could not find container \"092d682533ef4dd4fce564dfead035598355d2c0e076d963e5b9852be346f785\": container with ID starting with 092d682533ef4dd4fce564dfead035598355d2c0e076d963e5b9852be346f785 not found: ID does not exist"
Nov 02 18:11:45.461729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.461527    4456 scope.go:122] "RemoveContainer" containerID="026fbcc527da33cce56513069ee46f158e36543f79011d0d0036bc53fe44819c"
Nov 02 18:11:45.462420 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:45.462067    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"026fbcc527da33cce56513069ee46f158e36543f79011d0d0036bc53fe44819c\": container with ID starting with 026fbcc527da33cce56513069ee46f158e36543f79011d0d0036bc53fe44819c not found: ID does not exist" containerID="026fbcc527da33cce56513069ee46f158e36543f79011d0d0036bc53fe44819c"
Nov 02 18:11:45.462420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.462099    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"026fbcc527da33cce56513069ee46f158e36543f79011d0d0036bc53fe44819c"} err="failed to get container status \"026fbcc527da33cce56513069ee46f158e36543f79011d0d0036bc53fe44819c\": rpc error: code = NotFound desc = could not find container \"026fbcc527da33cce56513069ee46f158e36543f79011d0d0036bc53fe44819c\": container with ID starting with 026fbcc527da33cce56513069ee46f158e36543f79011d0d0036bc53fe44819c not found: ID does not exist"
Nov 02 18:11:45.462420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.462123    4456 scope.go:122] "RemoveContainer" containerID="b40a2aae2b6ef5d220e6c8a56acd93bfa30d09d873cc836d366f512f7dc799f8"
Nov 02 18:11:45.498586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.480714    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-workload-certs\") pod \"73849b3d-673e-41a8-aff1-db9f0a9cce85\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") "
Nov 02 18:11:45.498586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.480833    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-data\") pod \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") "
Nov 02 18:11:45.498586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.480881    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-credential-socket\") pod \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") "
Nov 02 18:11:45.498586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.480920    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-credential-socket\") pod \"73849b3d-673e-41a8-aff1-db9f0a9cce85\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") "
Nov 02 18:11:45.498586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.480967    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-token\" (UniqueName: \"kubernetes.io/projected/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-token\") pod \"73849b3d-673e-41a8-aff1-db9f0a9cce85\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") "
Nov 02 18:11:45.499068 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.481003    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istiod-ca-cert\") pod \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") "
Nov 02 18:11:45.499068 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.481048    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-workload-socket\") pod \"73849b3d-673e-41a8-aff1-db9f0a9cce85\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") "
Nov 02 18:11:45.499068 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.481098    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-token\" (UniqueName: \"kubernetes.io/projected/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-token\") pod \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") "
Nov 02 18:11:45.499068 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.481132    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-podinfo\") pod \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") "
Nov 02 18:11:45.499068 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.481167    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-podinfo\") pod \"73849b3d-673e-41a8-aff1-db9f0a9cce85\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") "
Nov 02 18:11:45.499304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.481229    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73849b3d-673e-41a8-aff1-db9f0a9cce85-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73849b3d-673e-41a8-aff1-db9f0a9cce85-istiod-ca-cert\") pod \"73849b3d-673e-41a8-aff1-db9f0a9cce85\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") "
Nov 02 18:11:45.499304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.481349    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-envoy\") pod \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") "
Nov 02 18:11:45.499304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.481394    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-envoy\") pod \"73849b3d-673e-41a8-aff1-db9f0a9cce85\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") "
Nov 02 18:11:45.499304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.481609    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-workload-certs\") pod \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") "
Nov 02 18:11:45.499304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.481791    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-workload-socket\") pod \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") "
Nov 02 18:11:45.499577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.481855    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-kube-api-access-b8h64\" (UniqueName: \"kubernetes.io/projected/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-kube-api-access-b8h64\") pod \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\" (UID: \"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d\") "
Nov 02 18:11:45.499577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.482051    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-data\") pod \"73849b3d-673e-41a8-aff1-db9f0a9cce85\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") "
Nov 02 18:11:45.499577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.482145    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73849b3d-673e-41a8-aff1-db9f0a9cce85-kube-api-access-gzxpq\" (UniqueName: \"kubernetes.io/projected/73849b3d-673e-41a8-aff1-db9f0a9cce85-kube-api-access-gzxpq\") pod \"73849b3d-673e-41a8-aff1-db9f0a9cce85\" (UID: \"73849b3d-673e-41a8-aff1-db9f0a9cce85\") "
Nov 02 18:11:45.499577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.493080    4456 scope.go:122] "RemoveContainer" containerID="b40a2aae2b6ef5d220e6c8a56acd93bfa30d09d873cc836d366f512f7dc799f8"
Nov 02 18:11:45.499577 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:45.493497    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b40a2aae2b6ef5d220e6c8a56acd93bfa30d09d873cc836d366f512f7dc799f8\": container with ID starting with b40a2aae2b6ef5d220e6c8a56acd93bfa30d09d873cc836d366f512f7dc799f8 not found: ID does not exist" containerID="b40a2aae2b6ef5d220e6c8a56acd93bfa30d09d873cc836d366f512f7dc799f8"
Nov 02 18:11:45.499800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.493581    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b40a2aae2b6ef5d220e6c8a56acd93bfa30d09d873cc836d366f512f7dc799f8"} err="failed to get container status \"b40a2aae2b6ef5d220e6c8a56acd93bfa30d09d873cc836d366f512f7dc799f8\": rpc error: code = NotFound desc = could not find container \"b40a2aae2b6ef5d220e6c8a56acd93bfa30d09d873cc836d366f512f7dc799f8\": container with ID starting with b40a2aae2b6ef5d220e6c8a56acd93bfa30d09d873cc836d366f512f7dc799f8 not found: ID does not exist"
Nov 02 18:11:45.499800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.493608    4456 scope.go:122] "RemoveContainer" containerID="9d4c1c0786ab2e7395cf39120539ef4b85c634872ec650a8b5de0096e6ff0bd0"
Nov 02 18:11:45.499800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.498467    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73849b3d-673e-41a8-aff1-db9f0a9cce85-istiod-ca-cert" pod "73849b3d-673e-41a8-aff1-db9f0a9cce85" (UID: "73849b3d-673e-41a8-aff1-db9f0a9cce85"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:45.500869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.500807    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-credential-socket" pod "73849b3d-673e-41a8-aff1-db9f0a9cce85" (UID: "73849b3d-673e-41a8-aff1-db9f0a9cce85"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.501437 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.501417    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-workload-certs" pod "73849b3d-673e-41a8-aff1-db9f0a9cce85" (UID: "73849b3d-673e-41a8-aff1-db9f0a9cce85"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.501963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.501946    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-data" pod "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" (UID: "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.502482 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.502465    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-credential-socket" pod "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" (UID: "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.503630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.503047    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istiod-ca-cert" pod "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" (UID: "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:45.504165 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.504142    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-workload-socket" pod "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" (UID: "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.504565 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.504536    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-workload-certs" pod "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" (UID: "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.510209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.510180    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-podinfo" pod "73849b3d-673e-41a8-aff1-db9f0a9cce85" (UID: "73849b3d-673e-41a8-aff1-db9f0a9cce85"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:45.515504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.514450    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/73849b3d-673e-41a8-aff1-db9f0a9cce85-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.515504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.514483    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.515504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.514498    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.515504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.514511    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.515504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.514526    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.515504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.514544    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.515504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.514558    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.515504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.514573    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.515894 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.514587    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.520314 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.519574    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-data" pod "73849b3d-673e-41a8-aff1-db9f0a9cce85" (UID: "73849b3d-673e-41a8-aff1-db9f0a9cce85"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.525829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.525787    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-workload-socket" pod "73849b3d-673e-41a8-aff1-db9f0a9cce85" (UID: "73849b3d-673e-41a8-aff1-db9f0a9cce85"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.546850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.539327    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-envoy" pod "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" (UID: "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.546850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.539681    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-kube-api-access-b8h64" pod "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" (UID: "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d"). InnerVolumeSpecName "kube-api-access-b8h64". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:45.546850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.546365    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-podinfo" pod "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" (UID: "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:45.546850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.546668    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"]
Nov 02 18:11:45.557022 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.555008    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-token" pod "73849b3d-673e-41a8-aff1-db9f0a9cce85" (UID: "73849b3d-673e-41a8-aff1-db9f0a9cce85"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:45.557022 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.555682    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-envoy" pod "73849b3d-673e-41a8-aff1-db9f0a9cce85" (UID: "73849b3d-673e-41a8-aff1-db9f0a9cce85"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.561131 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.560435    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-5xkqq"]
Nov 02 18:11:45.569594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.569566    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-token" pod "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" (UID: "cb09170f-1b12-4a26-8e5b-c5e2c2964a4d"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:45.570635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.570596    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:45.579851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.579461    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73849b3d-673e-41a8-aff1-db9f0a9cce85-kube-api-access-gzxpq" pod "73849b3d-673e-41a8-aff1-db9f0a9cce85" (UID: "73849b3d-673e-41a8-aff1-db9f0a9cce85"). InnerVolumeSpecName "kube-api-access-gzxpq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:45.587969 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.587944    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"]
Nov 02 18:11:45.590579 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.590501    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 18:11:45.599042 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.597031    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-d5b905-openshift-default-775cd76ff-vpnr8"]
Nov 02 18:11:45.620706 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.620674    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.620706 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.620708    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.620933 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.620723    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b8h64\" (UniqueName: \"kubernetes.io/projected/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-kube-api-access-b8h64\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.620933 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.620739    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.620933 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.620753    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gzxpq\" (UniqueName: \"kubernetes.io/projected/73849b3d-673e-41a8-aff1-db9f0a9cce85-kube-api-access-gzxpq\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.620933 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.620763    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/73849b3d-673e-41a8-aff1-db9f0a9cce85-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.620933 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.620775    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/73849b3d-673e-41a8-aff1-db9f0a9cce85-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.620933 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.620786    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.620933 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.620798    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.627335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.627308    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" path="/var/lib/kubelet/pods/408afe2d-75a0-4537-b58c-9ab31f161685/volumes"
Nov 02 18:11:45.628183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.628163    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" path="/var/lib/kubelet/pods/6aa41ecd-4a93-4b17-8e84-3f3c018c4d21/volumes"
Nov 02 18:11:45.628944 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.628925    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" path="/var/lib/kubelet/pods/778b9a36-6a7b-4251-a91a-999e358d8c5c/volumes"
Nov 02 18:11:45.630722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.630699    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ef6443b3-b452-4868-9cde-f762cf522284" path="/var/lib/kubelet/pods/ef6443b3-b452-4868-9cde-f762cf522284/volumes"
Nov 02 18:11:45.671490 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.671468    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 18:11:45.705576 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.705550    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:11:45.733841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.729651    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-workload-certs\") pod \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") "
Nov 02 18:11:45.733841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.729784    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-data\") pod \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") "
Nov 02 18:11:45.733841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.729861    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-credential-socket\") pod \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") "
Nov 02 18:11:45.733841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.729930    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb882f6-fd8f-46ae-9396-b52498c5d9db-kube-api-access-ccvc9\" (UniqueName: \"kubernetes.io/projected/5bb882f6-fd8f-46ae-9396-b52498c5d9db-kube-api-access-ccvc9\") pod \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") "
Nov 02 18:11:45.733841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.730071    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-envoy\") pod \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") "
Nov 02 18:11:45.734241 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.730216    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-token\" (UniqueName: \"kubernetes.io/projected/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-token\") pod \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") "
Nov 02 18:11:45.734241 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.730328    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-workload-socket\") pod \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") "
Nov 02 18:11:45.734241 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.730404    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istiod-ca-cert\") pod \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") "
Nov 02 18:11:45.734241 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.730532    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-podinfo\") pod \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\" (UID: \"5bb882f6-fd8f-46ae-9396-b52498c5d9db\") "
Nov 02 18:11:45.792154 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.791909    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-data" pod "5bb882f6-fd8f-46ae-9396-b52498c5d9db" (UID: "5bb882f6-fd8f-46ae-9396-b52498c5d9db"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.795912 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.792370    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-workload-certs" pod "5bb882f6-fd8f-46ae-9396-b52498c5d9db" (UID: "5bb882f6-fd8f-46ae-9396-b52498c5d9db"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.795912 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.793891    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-workload-socket" pod "5bb882f6-fd8f-46ae-9396-b52498c5d9db" (UID: "5bb882f6-fd8f-46ae-9396-b52498c5d9db"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.795912 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.794302    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-credential-socket" pod "5bb882f6-fd8f-46ae-9396-b52498c5d9db" (UID: "5bb882f6-fd8f-46ae-9396-b52498c5d9db"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.796338 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.796299    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istiod-ca-cert" pod "5bb882f6-fd8f-46ae-9396-b52498c5d9db" (UID: "5bb882f6-fd8f-46ae-9396-b52498c5d9db"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:45.807089 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.806990    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-podinfo" pod "5bb882f6-fd8f-46ae-9396-b52498c5d9db" (UID: "5bb882f6-fd8f-46ae-9396-b52498c5d9db"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:45.834578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.833362    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-kube-api-access-8w2nc\" (UniqueName: \"kubernetes.io/projected/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-kube-api-access-8w2nc\") pod \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") "
Nov 02 18:11:45.834578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.833422    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-workload-socket\") pod \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") "
Nov 02 18:11:45.834578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.833457    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istiod-ca-cert\") pod \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") "
Nov 02 18:11:45.834578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.833528    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/999df778-cd9b-4b41-852d-4661a7326483-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999df778-cd9b-4b41-852d-4661a7326483-istio-podinfo\") pod \"999df778-cd9b-4b41-852d-4661a7326483\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") "
Nov 02 18:11:45.834578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.833615    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-workload-certs\") pod \"999df778-cd9b-4b41-852d-4661a7326483\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") "
Nov 02 18:11:45.834973 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.833648    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-podinfo\") pod \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") "
Nov 02 18:11:45.834973 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.833723    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/999df778-cd9b-4b41-852d-4661a7326483-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999df778-cd9b-4b41-852d-4661a7326483-istiod-ca-cert\") pod \"999df778-cd9b-4b41-852d-4661a7326483\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") "
Nov 02 18:11:45.834973 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.833749    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-workload-socket\") pod \"999df778-cd9b-4b41-852d-4661a7326483\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") "
Nov 02 18:11:45.834973 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.833788    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-istio-data\") pod \"999df778-cd9b-4b41-852d-4661a7326483\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") "
Nov 02 18:11:45.834973 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.833854    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999df778-cd9b-4b41-852d-4661a7326483-kube-api-access-q2gdq\" (UniqueName: \"kubernetes.io/projected/999df778-cd9b-4b41-852d-4661a7326483-kube-api-access-q2gdq\") pod \"999df778-cd9b-4b41-852d-4661a7326483\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") "
Nov 02 18:11:45.835205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.833931    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-credential-socket\") pod \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") "
Nov 02 18:11:45.835205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.834024    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-data\") pod \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") "
Nov 02 18:11:45.835205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.834066    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-credential-socket\") pod \"999df778-cd9b-4b41-852d-4661a7326483\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") "
Nov 02 18:11:45.835205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.834103    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-token\" (UniqueName: \"kubernetes.io/projected/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-token\") pod \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") "
Nov 02 18:11:45.835205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.834140    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/999df778-cd9b-4b41-852d-4661a7326483-istio-token\" (UniqueName: \"kubernetes.io/projected/999df778-cd9b-4b41-852d-4661a7326483-istio-token\") pod \"999df778-cd9b-4b41-852d-4661a7326483\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") "
Nov 02 18:11:45.835420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.834556    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/999df778-cd9b-4b41-852d-4661a7326483-istiod-ca-cert" pod "999df778-cd9b-4b41-852d-4661a7326483" (UID: "999df778-cd9b-4b41-852d-4661a7326483"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:45.835420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.834627    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-istio-envoy\") pod \"999df778-cd9b-4b41-852d-4661a7326483\" (UID: \"999df778-cd9b-4b41-852d-4661a7326483\") "
Nov 02 18:11:45.835420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.834697    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-envoy\") pod \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") "
Nov 02 18:11:45.835420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.834842    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-workload-certs\") pod \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\" (UID: \"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3\") "
Nov 02 18:11:45.835420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.834967    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb882f6-fd8f-46ae-9396-b52498c5d9db-kube-api-access-ccvc9" pod "5bb882f6-fd8f-46ae-9396-b52498c5d9db" (UID: "5bb882f6-fd8f-46ae-9396-b52498c5d9db"). InnerVolumeSpecName "kube-api-access-ccvc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:45.835607 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.835287    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-credential-socket" pod "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" (UID: "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.843674 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.837900    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-workload-socket" pod "999df778-cd9b-4b41-852d-4661a7326483" (UID: "999df778-cd9b-4b41-852d-4661a7326483"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.844238 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.844122    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-workload-certs" pod "999df778-cd9b-4b41-852d-4661a7326483" (UID: "999df778-cd9b-4b41-852d-4661a7326483"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.848376 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.844410    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-data" pod "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" (UID: "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.848376 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.844626    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-credential-socket" pod "999df778-cd9b-4b41-852d-4661a7326483" (UID: "999df778-cd9b-4b41-852d-4661a7326483"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.848376 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.845948    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-istio-data" pod "999df778-cd9b-4b41-852d-4661a7326483" (UID: "999df778-cd9b-4b41-852d-4661a7326483"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.862201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.862167    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-workload-socket" pod "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" (UID: "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.865917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.864991    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istiod-ca-cert" pod "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" (UID: "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:45.876504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.876171    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-workload-certs" pod "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" (UID: "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.881959 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.881933    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.886973 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.886933    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.935292 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.911868    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-envoy" pod "5bb882f6-fd8f-46ae-9396-b52498c5d9db" (UID: "5bb882f6-fd8f-46ae-9396-b52498c5d9db"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.935477 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.911908    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-token" pod "5bb882f6-fd8f-46ae-9396-b52498c5d9db" (UID: "5bb882f6-fd8f-46ae-9396-b52498c5d9db"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:45.935576 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.930001    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/999df778-cd9b-4b41-852d-4661a7326483-istio-podinfo" pod "999df778-cd9b-4b41-852d-4661a7326483" (UID: "999df778-cd9b-4b41-852d-4661a7326483"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:45.935655 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.935239    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccvc9\" (UniqueName: \"kubernetes.io/projected/5bb882f6-fd8f-46ae-9396-b52498c5d9db-kube-api-access-ccvc9\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.935743 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.935726    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.935867 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.935853    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.935949 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.935938    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.936029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.936020    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.936106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.936095    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.936174 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.936165    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.936249 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.936239    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.936319 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.936308    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.936394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.936385    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.936469 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.936459    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.936548 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.936535    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/999df778-cd9b-4b41-852d-4661a7326483-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.936629 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.936614    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.936709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.936699    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.936782 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.936773    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:45.954454 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.954393    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999df778-cd9b-4b41-852d-4661a7326483-istio-token" pod "999df778-cd9b-4b41-852d-4661a7326483" (UID: "999df778-cd9b-4b41-852d-4661a7326483"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:45.955721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.954669    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-envoy" pod "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" (UID: "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.955721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.954761    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-kube-api-access-8w2nc" pod "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" (UID: "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3"). InnerVolumeSpecName "kube-api-access-8w2nc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:45.955721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.954857    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-istio-envoy" pod "999df778-cd9b-4b41-852d-4661a7326483" (UID: "999df778-cd9b-4b41-852d-4661a7326483"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:45.955721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.954927    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-token" pod "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" (UID: "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:45.955721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.954975    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-podinfo" pod "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" (UID: "694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:45.956225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.956044    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/999df778-cd9b-4b41-852d-4661a7326483-kube-api-access-q2gdq" pod "999df778-cd9b-4b41-852d-4661a7326483" (UID: "999df778-cd9b-4b41-852d-4661a7326483"). InnerVolumeSpecName "kube-api-access-q2gdq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:45.997597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:45.997562    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:46.052247 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.052212    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:46.052758 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.052735    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q2gdq\" (UniqueName: \"kubernetes.io/projected/999df778-cd9b-4b41-852d-4661a7326483-kube-api-access-q2gdq\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:46.052937 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.052918    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:46.053030 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.053019    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:46.053122 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.053111    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/999df778-cd9b-4b41-852d-4661a7326483-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:46.053201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.053190    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/999df778-cd9b-4b41-852d-4661a7326483-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:46.053279 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.053269    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:46.053364 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.053352    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5bb882f6-fd8f-46ae-9396-b52498c5d9db-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:46.053465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.053453    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8w2nc\" (UniqueName: \"kubernetes.io/projected/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3-kube-api-access-8w2nc\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:46.053544 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.053532    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/999df778-cd9b-4b41-852d-4661a7326483-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:46.149707 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.149618    4456 generic.go:356] "Generic (PLEG): container finished" podID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerID="8fd11b5da302643890d03cf8b90f90991b0ace49793a57bde1469d40e28383be" exitCode=0
Nov 02 18:11:46.149707 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.149686    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"
Nov 02 18:11:46.149950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.149706    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" event={"ID":"5bb882f6-fd8f-46ae-9396-b52498c5d9db","Type":"ContainerDied","Data":"8fd11b5da302643890d03cf8b90f90991b0ace49793a57bde1469d40e28383be"}
Nov 02 18:11:46.150021 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.150001    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s" event={"ID":"5bb882f6-fd8f-46ae-9396-b52498c5d9db","Type":"ContainerDied","Data":"141a56e13647c456e1524d4bceb0ae64ac8bb6c75d433cd9258439496b0d9621"}
Nov 02 18:11:46.150072 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.150039    4456 scope.go:122] "RemoveContainer" containerID="8fd11b5da302643890d03cf8b90f90991b0ace49793a57bde1469d40e28383be"
Nov 02 18:11:46.153526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.153502    4456 generic.go:356] "Generic (PLEG): container finished" podID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerID="8f711850377e64150821c64844dc9b4f619291b996bda5ba3ee277096962757c" exitCode=0
Nov 02 18:11:46.153686 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.153539    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" event={"ID":"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3","Type":"ContainerDied","Data":"8f711850377e64150821c64844dc9b4f619291b996bda5ba3ee277096962757c"}
Nov 02 18:11:46.153868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.153802    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck" event={"ID":"694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3","Type":"ContainerDied","Data":"a6ef42ab6840fc568f9b0249306f579e85614adaade0478bce59dcb673d5d337"}
Nov 02 18:11:46.154056 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.154044    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"
Nov 02 18:11:46.159192 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.157488    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88" event={"ID":"cb09170f-1b12-4a26-8e5b-c5e2c2964a4d","Type":"ContainerDied","Data":"aa577c73e9d0c2359c23f08a768cebf7f5e5f967253d141aa0c98378d78320c0"}
Nov 02 18:11:46.159192 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.157602    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"
Nov 02 18:11:46.164684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.164263    4456 generic.go:356] "Generic (PLEG): container finished" podID="999df778-cd9b-4b41-852d-4661a7326483" containerID="c492b110991299865769d96c519b16a0cdfd155200b1d4b35cdb7daa5de888f7" exitCode=0
Nov 02 18:11:46.164684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.164387    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" event={"ID":"999df778-cd9b-4b41-852d-4661a7326483","Type":"ContainerDied","Data":"c492b110991299865769d96c519b16a0cdfd155200b1d4b35cdb7daa5de888f7"}
Nov 02 18:11:46.164684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.164444    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf" event={"ID":"999df778-cd9b-4b41-852d-4661a7326483","Type":"ContainerDied","Data":"e8ac58feb06de46a148ffed2ba94bf93173c751ea50dba769ff13cd0fe67b9fb"}
Nov 02 18:11:46.164684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.164575    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"
Nov 02 18:11:46.169084 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.167850    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7" event={"ID":"73849b3d-673e-41a8-aff1-db9f0a9cce85","Type":"ContainerDied","Data":"d1f60cf1a33f8ea16b7853b8f00764dcc02533e84f1f4e0ed57c2eb6b4b3585a"}
Nov 02 18:11:46.169084 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.167960    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"
Nov 02 18:11:46.169084 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.168661    4456 scope.go:122] "RemoveContainer" containerID="8fd11b5da302643890d03cf8b90f90991b0ace49793a57bde1469d40e28383be"
Nov 02 18:11:46.169459 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:46.169329    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8fd11b5da302643890d03cf8b90f90991b0ace49793a57bde1469d40e28383be\": container with ID starting with 8fd11b5da302643890d03cf8b90f90991b0ace49793a57bde1469d40e28383be not found: ID does not exist" containerID="8fd11b5da302643890d03cf8b90f90991b0ace49793a57bde1469d40e28383be"
Nov 02 18:11:46.169459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.169369    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8fd11b5da302643890d03cf8b90f90991b0ace49793a57bde1469d40e28383be"} err="failed to get container status \"8fd11b5da302643890d03cf8b90f90991b0ace49793a57bde1469d40e28383be\": rpc error: code = NotFound desc = could not find container \"8fd11b5da302643890d03cf8b90f90991b0ace49793a57bde1469d40e28383be\": container with ID starting with 8fd11b5da302643890d03cf8b90f90991b0ace49793a57bde1469d40e28383be not found: ID does not exist"
Nov 02 18:11:46.169459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.169391    4456 scope.go:122] "RemoveContainer" containerID="8f711850377e64150821c64844dc9b4f619291b996bda5ba3ee277096962757c"
Nov 02 18:11:46.184329 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:46.184297    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:11:46.189121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.189101    4456 scope.go:122] "RemoveContainer" containerID="8f711850377e64150821c64844dc9b4f619291b996bda5ba3ee277096962757c"
Nov 02 18:11:46.189464 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:46.189440    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8f711850377e64150821c64844dc9b4f619291b996bda5ba3ee277096962757c\": container with ID starting with 8f711850377e64150821c64844dc9b4f619291b996bda5ba3ee277096962757c not found: ID does not exist" containerID="8f711850377e64150821c64844dc9b4f619291b996bda5ba3ee277096962757c"
Nov 02 18:11:46.189548 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.189476    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8f711850377e64150821c64844dc9b4f619291b996bda5ba3ee277096962757c"} err="failed to get container status \"8f711850377e64150821c64844dc9b4f619291b996bda5ba3ee277096962757c\": rpc error: code = NotFound desc = could not find container \"8f711850377e64150821c64844dc9b4f619291b996bda5ba3ee277096962757c\": container with ID starting with 8f711850377e64150821c64844dc9b4f619291b996bda5ba3ee277096962757c not found: ID does not exist"
Nov 02 18:11:46.189548 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.189500    4456 scope.go:122] "RemoveContainer" containerID="33774b00458e580d97ee67142da150ad8d47b01d7485185ee8fef16f91745dda"
Nov 02 18:11:46.231287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.231253    4456 scope.go:122] "RemoveContainer" containerID="c492b110991299865769d96c519b16a0cdfd155200b1d4b35cdb7daa5de888f7"
Nov 02 18:11:46.244121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.244104    4456 scope.go:122] "RemoveContainer" containerID="1f92044f46d783f762d4c677bea78e3b2f2e6ab30bddaea987c9b15454088386"
Nov 02 18:11:46.267288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.267247    4456 scope.go:122] "RemoveContainer" containerID="c492b110991299865769d96c519b16a0cdfd155200b1d4b35cdb7daa5de888f7"
Nov 02 18:11:46.268967 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:46.268771    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"c492b110991299865769d96c519b16a0cdfd155200b1d4b35cdb7daa5de888f7\": container with ID starting with c492b110991299865769d96c519b16a0cdfd155200b1d4b35cdb7daa5de888f7 not found: ID does not exist" containerID="c492b110991299865769d96c519b16a0cdfd155200b1d4b35cdb7daa5de888f7"
Nov 02 18:11:46.268967 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.268856    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"c492b110991299865769d96c519b16a0cdfd155200b1d4b35cdb7daa5de888f7"} err="failed to get container status \"c492b110991299865769d96c519b16a0cdfd155200b1d4b35cdb7daa5de888f7\": rpc error: code = NotFound desc = could not find container \"c492b110991299865769d96c519b16a0cdfd155200b1d4b35cdb7daa5de888f7\": container with ID starting with c492b110991299865769d96c519b16a0cdfd155200b1d4b35cdb7daa5de888f7 not found: ID does not exist"
Nov 02 18:11:46.268967 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.268883    4456 scope.go:122] "RemoveContainer" containerID="1f92044f46d783f762d4c677bea78e3b2f2e6ab30bddaea987c9b15454088386"
Nov 02 18:11:46.269488 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:46.269461    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1f92044f46d783f762d4c677bea78e3b2f2e6ab30bddaea987c9b15454088386\": container with ID starting with 1f92044f46d783f762d4c677bea78e3b2f2e6ab30bddaea987c9b15454088386 not found: ID does not exist" containerID="1f92044f46d783f762d4c677bea78e3b2f2e6ab30bddaea987c9b15454088386"
Nov 02 18:11:46.269580 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.269498    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1f92044f46d783f762d4c677bea78e3b2f2e6ab30bddaea987c9b15454088386"} err="failed to get container status \"1f92044f46d783f762d4c677bea78e3b2f2e6ab30bddaea987c9b15454088386\": rpc error: code = NotFound desc = could not find container \"1f92044f46d783f762d4c677bea78e3b2f2e6ab30bddaea987c9b15454088386\": container with ID starting with 1f92044f46d783f762d4c677bea78e3b2f2e6ab30bddaea987c9b15454088386 not found: ID does not exist"
Nov 02 18:11:46.269580 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.269519    4456 scope.go:122] "RemoveContainer" containerID="321f32c6b20dd4c469c4740acf6ac58556d8ab5567e481fbe7905842a155c1f0"
Nov 02 18:11:46.273478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.273454    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"]
Nov 02 18:11:46.279226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.279201    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-2jq88"]
Nov 02 18:11:46.330758 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.330726    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"]
Nov 02 18:11:46.335184 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.335158    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-qp2ck"]
Nov 02 18:11:46.372193 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.372168    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"]
Nov 02 18:11:46.375920 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.375900    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-p495s"]
Nov 02 18:11:46.415985 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.415906    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"]
Nov 02 18:11:46.420741 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.420714    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-5fd3e0-openshift-default-77476c6cbc-vwvnf"]
Nov 02 18:11:46.465838 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.465771    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"]
Nov 02 18:11:46.478237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.478184    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-2c9b1d-openshift-default-84ddbc878c-dq2n7"]
Nov 02 18:11:46.660652 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.660621    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"]
Nov 02 18:11:46.660872 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.660845    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy" containerID="cri-o://97a86bee14d24f04dff0a2c6c82d05fab51870580f831e892bca987e63a5078c" gracePeriod=30
Nov 02 18:11:46.668248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.667604    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"]
Nov 02 18:11:46.668248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.668015    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" containerID="cri-o://34675a9bb3e0d78cbe38155c7ebf2cd25bd2500003af0cd5b590e6ab4a83d61a" gracePeriod=30
Nov 02 18:11:46.673742 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.673714    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"]
Nov 02 18:11:46.674178 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.674019    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy" containerID="cri-o://6f672ff50b415c9f5d877d39fc43cbb4be0f8cf802d9f572dcc1a950321d5ed3" gracePeriod=30
Nov 02 18:11:46.699883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.679572    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"]
Nov 02 18:11:46.699883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.685840    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"]
Nov 02 18:11:46.699883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.686035    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy" containerID="cri-o://4cb6844084f3272f4de73238806060cd7601c7e4e7b1a56e862c1e414d1e1d67" gracePeriod=30
Nov 02 18:11:46.699883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.691443    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"]
Nov 02 18:11:46.699883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.691627    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" containerID="cri-o://7ea8d9afddf0f6e4f043e8c148239bb0b12e1d9cf9c2338a03073a647dc99771" gracePeriod=30
Nov 02 18:11:46.699883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.695894    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" containerID="cri-o://7d77b8d6127469706b8666ac8056de7ddc308b1c2fb37e9a35ec932c0e735f54" gracePeriod=30
Nov 02 18:11:46.702872 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.702185    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"]
Nov 02 18:11:46.702872 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.702380    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" containerID="cri-o://95adba9c28b54d3f0298de1dec2292a9810fd828ad712b762f15dc288bbab452" gracePeriod=30
Nov 02 18:11:46.812235 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.812204    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"]
Nov 02 18:11:46.812508 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.812467    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy" containerID="cri-o://730da757c8ffa0ec7438177532c8a6ce99b4b9d761e88b8e9e1bb45179e1eca6" gracePeriod=30
Nov 02 18:11:46.821968 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.821926    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"]
Nov 02 18:11:46.822216 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.822192    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" podUID="0f2089f4-8ec7-45ce-a2df-0bf024b4158f" containerName="discovery" containerID="cri-o://593b4317838a7e992136f4e9e542b750536a8fd59cf1ad26c128305c332da9be" gracePeriod=30
Nov 02 18:11:46.865978 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.865946    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"]
Nov 02 18:11:46.866449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.866385    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy" containerID="cri-o://60addef8887b57cc6daa92f4b0dbb1eb9ed3aa5f9e9c2ecde7a1e071117ccff8" gracePeriod=30
Nov 02 18:11:46.871482 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.871455    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"]
Nov 02 18:11:46.871692 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.871670    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy" containerID="cri-o://aff8684245e6811977d173dba51890ae99616b8073fb04935c6a2893e1dadaf3" gracePeriod=30
Nov 02 18:11:46.918435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.918357    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"]
Nov 02 18:11:46.918758 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.918735    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy" containerID="cri-o://99a6168054233269cb6e8344874ad06a145a87f2aae8004e7910f961ede56963" gracePeriod=30
Nov 02 18:11:46.964996 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.964956    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"]
Nov 02 18:11:46.965230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:46.965201    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerName="istio-proxy" containerID="cri-o://aaaa12bbfe6411dd90db578524e8a9702da102f37573e9977362c806a70e72b3" gracePeriod=30
Nov 02 18:11:47.166628 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.166605    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 18:11:47.184750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.184719    4456 generic.go:356] "Generic (PLEG): container finished" podID="0f2089f4-8ec7-45ce-a2df-0bf024b4158f" containerID="593b4317838a7e992136f4e9e542b750536a8fd59cf1ad26c128305c332da9be" exitCode=0
Nov 02 18:11:47.184915 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.184761    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"
Nov 02 18:11:47.184915 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.184868    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" event={"ID":"0f2089f4-8ec7-45ce-a2df-0bf024b4158f","Type":"ContainerDied","Data":"593b4317838a7e992136f4e9e542b750536a8fd59cf1ad26c128305c332da9be"}
Nov 02 18:11:47.184915 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.184909    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82" event={"ID":"0f2089f4-8ec7-45ce-a2df-0bf024b4158f","Type":"ContainerDied","Data":"e8c411f43f20cb4752201faa744662d6d835f0af36906442b13f55878b99bce8"}
Nov 02 18:11:47.185088 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.184937    4456 scope.go:122] "RemoveContainer" containerID="593b4317838a7e992136f4e9e542b750536a8fd59cf1ad26c128305c332da9be"
Nov 02 18:11:47.199071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.199027    4456 scope.go:122] "RemoveContainer" containerID="593b4317838a7e992136f4e9e542b750536a8fd59cf1ad26c128305c332da9be"
Nov 02 18:11:47.199418 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:47.199391    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"593b4317838a7e992136f4e9e542b750536a8fd59cf1ad26c128305c332da9be\": container with ID starting with 593b4317838a7e992136f4e9e542b750536a8fd59cf1ad26c128305c332da9be not found: ID does not exist" containerID="593b4317838a7e992136f4e9e542b750536a8fd59cf1ad26c128305c332da9be"
Nov 02 18:11:47.199520 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.199427    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"593b4317838a7e992136f4e9e542b750536a8fd59cf1ad26c128305c332da9be"} err="failed to get container status \"593b4317838a7e992136f4e9e542b750536a8fd59cf1ad26c128305c332da9be\": rpc error: code = NotFound desc = could not find container \"593b4317838a7e992136f4e9e542b750536a8fd59cf1ad26c128305c332da9be\": container with ID starting with 593b4317838a7e992136f4e9e542b750536a8fd59cf1ad26c128305c332da9be not found: ID does not exist"
Nov 02 18:11:47.307575 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.307544    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-kube-api-access-sfntx\" (UniqueName: \"kubernetes.io/projected/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-kube-api-access-sfntx\") pod \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") "
Nov 02 18:11:47.307762 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.307625    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-local-certs\") pod \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") "
Nov 02 18:11:47.307762 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.307659    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-cacerts\" (UniqueName: \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-cacerts\") pod \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") "
Nov 02 18:11:47.307762 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.307686    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-kubeconfig\") pod \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") "
Nov 02 18:11:47.307762 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.307741    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-csr-dns-cert\") pod \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") "
Nov 02 18:11:47.307964 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.307840    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-csr-ca-configmap\") pod \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") "
Nov 02 18:11:47.307964 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.307908    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-token\" (UniqueName: \"kubernetes.io/projected/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-token\") pod \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\" (UID: \"0f2089f4-8ec7-45ce-a2df-0bf024b4158f\") "
Nov 02 18:11:47.310702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.310667    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-csr-ca-configmap" pod "0f2089f4-8ec7-45ce-a2df-0bf024b4158f" (UID: "0f2089f4-8ec7-45ce-a2df-0bf024b4158f"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:47.312465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.312435    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-kubeconfig" pod "0f2089f4-8ec7-45ce-a2df-0bf024b4158f" (UID: "0f2089f4-8ec7-45ce-a2df-0bf024b4158f"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:11:47.315876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.314619    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-token" pod "0f2089f4-8ec7-45ce-a2df-0bf024b4158f" (UID: "0f2089f4-8ec7-45ce-a2df-0bf024b4158f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:47.315876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.315235    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-csr-dns-cert" pod "0f2089f4-8ec7-45ce-a2df-0bf024b4158f" (UID: "0f2089f4-8ec7-45ce-a2df-0bf024b4158f"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:11:47.315876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.315365    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-kube-api-access-sfntx" pod "0f2089f4-8ec7-45ce-a2df-0bf024b4158f" (UID: "0f2089f4-8ec7-45ce-a2df-0bf024b4158f"). InnerVolumeSpecName "kube-api-access-sfntx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:47.334596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.329644    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-local-certs" pod "0f2089f4-8ec7-45ce-a2df-0bf024b4158f" (UID: "0f2089f4-8ec7-45ce-a2df-0bf024b4158f"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:47.334596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.329693    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-cacerts" pod "0f2089f4-8ec7-45ce-a2df-0bf024b4158f" (UID: "0f2089f4-8ec7-45ce-a2df-0bf024b4158f"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:11:47.410186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.410150    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sfntx\" (UniqueName: \"kubernetes.io/projected/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-kube-api-access-sfntx\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:47.410186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.410183    4456 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-local-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:47.410186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.410192    4456 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-cacerts\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:47.410496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.410201    4456 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-kubeconfig\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:47.410496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.410209    4456 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-csr-dns-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:47.410496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.410217    4456 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-csr-ca-configmap\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:47.410496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.410229    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/0f2089f4-8ec7-45ce-a2df-0bf024b4158f-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:47.546095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.546063    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"]
Nov 02 18:11:47.549975 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.549933    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-tdx82"]
Nov 02 18:11:47.618728 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.618694    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0f2089f4-8ec7-45ce-a2df-0bf024b4158f" path="/var/lib/kubelet/pods/0f2089f4-8ec7-45ce-a2df-0bf024b4158f/volumes"
Nov 02 18:11:47.619276 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.619258    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" path="/var/lib/kubelet/pods/5bb882f6-fd8f-46ae-9396-b52498c5d9db/volumes"
Nov 02 18:11:47.619705 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.619689    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" path="/var/lib/kubelet/pods/694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3/volumes"
Nov 02 18:11:47.620518 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.620498    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" path="/var/lib/kubelet/pods/73849b3d-673e-41a8-aff1-db9f0a9cce85/volumes"
Nov 02 18:11:47.620996 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.620980    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="999df778-cd9b-4b41-852d-4661a7326483" path="/var/lib/kubelet/pods/999df778-cd9b-4b41-852d-4661a7326483/volumes"
Nov 02 18:11:47.621436 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.621422    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" path="/var/lib/kubelet/pods/cb09170f-1b12-4a26-8e5b-c5e2c2964a4d/volumes"
Nov 02 18:11:47.890216 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:47.890073    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:48.303320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:48.303276    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:48.559685 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:48.559595    4456 patch_prober.go:29] interesting pod/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=upstream connect error or disconnect/reset before headers. reset reason: remote connection failure, transport failure reason: delayed connect error: Connection refused
Nov 02 18:11:48.559685 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:48.559662    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:11:49.688497 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:49.688454    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:49.975472 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:49.975383    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" podUID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" containerName="payload-processing" containerID="cri-o://ab3b1839843faf727063beb731cd4336c7b223cdec36d36195c04bd9689f1da3" gracePeriod=25
Nov 02 18:11:50.029694 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.029662    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:11:50.029853 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.029739    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:11:50.224859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.224799    4456 generic.go:356] "Generic (PLEG): container finished" podID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" containerID="ab3b1839843faf727063beb731cd4336c7b223cdec36d36195c04bd9689f1da3" exitCode=0
Nov 02 18:11:50.225016 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.224905    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" event={"ID":"c0ff0cbb-4acf-4d63-9579-6aeac659b2ea","Type":"ContainerDied","Data":"ab3b1839843faf727063beb731cd4336c7b223cdec36d36195c04bd9689f1da3"}
Nov 02 18:11:50.273599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.273575    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"
Nov 02 18:11:50.459417 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.459382    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c0ff0cbb-4acf-4d63-9579-6aeac659b2ea-kube-api-access-6phtq\" (UniqueName: \"kubernetes.io/projected/c0ff0cbb-4acf-4d63-9579-6aeac659b2ea-kube-api-access-6phtq\") pod \"c0ff0cbb-4acf-4d63-9579-6aeac659b2ea\" (UID: \"c0ff0cbb-4acf-4d63-9579-6aeac659b2ea\") "
Nov 02 18:11:50.459592 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.459438    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c0ff0cbb-4acf-4d63-9579-6aeac659b2ea-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ff0cbb-4acf-4d63-9579-6aeac659b2ea-plugins-config-volume\") pod \"c0ff0cbb-4acf-4d63-9579-6aeac659b2ea\" (UID: \"c0ff0cbb-4acf-4d63-9579-6aeac659b2ea\") "
Nov 02 18:11:50.459854 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.459825    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c0ff0cbb-4acf-4d63-9579-6aeac659b2ea-plugins-config-volume" pod "c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" (UID: "c0ff0cbb-4acf-4d63-9579-6aeac659b2ea"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:50.460041 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.460026    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c0ff0cbb-4acf-4d63-9579-6aeac659b2ea-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:50.461727 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.461702    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c0ff0cbb-4acf-4d63-9579-6aeac659b2ea-kube-api-access-6phtq" pod "c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" (UID: "c0ff0cbb-4acf-4d63-9579-6aeac659b2ea"). InnerVolumeSpecName "kube-api-access-6phtq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:50.476124 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476098    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"]
Nov 02 18:11:50.476716 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476703    4456 cpu_manager.go:414] "containerMap: removing container" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy"
Nov 02 18:11:50.476750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476718    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy"
Nov 02 18:11:50.476750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476731    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f54bd962-e719-41c4-9c28-56460e46d0f6" containerName="revoke-keys"
Nov 02 18:11:50.476750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476736    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f54bd962-e719-41c4-9c28-56460e46d0f6" containerName="revoke-keys"
Nov 02 18:11:50.476859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476765    4456 cpu_manager.go:414] "containerMap: removing container" podUID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" containerName="payload-processing"
Nov 02 18:11:50.476859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476770    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" containerName="payload-processing"
Nov 02 18:11:50.476859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476777    4456 cpu_manager.go:414] "containerMap: removing container" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy"
Nov 02 18:11:50.476859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476781    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy"
Nov 02 18:11:50.476859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476791    4456 cpu_manager.go:414] "containerMap: removing container" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy"
Nov 02 18:11:50.476859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476796    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy"
Nov 02 18:11:50.476859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476805    4456 cpu_manager.go:414] "containerMap: removing container" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy"
Nov 02 18:11:50.476859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476809    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy"
Nov 02 18:11:50.476859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476840    4456 cpu_manager.go:414] "containerMap: removing container" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy"
Nov 02 18:11:50.476859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476846    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy"
Nov 02 18:11:50.476859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476852    4456 cpu_manager.go:414] "containerMap: removing container" podUID="0f2089f4-8ec7-45ce-a2df-0bf024b4158f" containerName="discovery"
Nov 02 18:11:50.477138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476857    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0f2089f4-8ec7-45ce-a2df-0bf024b4158f" containerName="discovery"
Nov 02 18:11:50.477138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476864    4456 cpu_manager.go:414] "containerMap: removing container" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy"
Nov 02 18:11:50.477138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476868    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy"
Nov 02 18:11:50.477138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476877    4456 cpu_manager.go:414] "containerMap: removing container" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy"
Nov 02 18:11:50.477138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476882    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy"
Nov 02 18:11:50.477138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476888    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d99092cc-b126-48db-8762-a4b8d64ad350" containerName="revoke-keys"
Nov 02 18:11:50.477138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476893    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d99092cc-b126-48db-8762-a4b8d64ad350" containerName="revoke-keys"
Nov 02 18:11:50.477138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476899    4456 cpu_manager.go:414] "containerMap: removing container" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy"
Nov 02 18:11:50.477138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476903    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy"
Nov 02 18:11:50.477138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476914    4456 cpu_manager.go:414] "containerMap: removing container" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy"
Nov 02 18:11:50.477138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476918    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy"
Nov 02 18:11:50.477138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476926    4456 cpu_manager.go:414] "containerMap: removing container" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy"
Nov 02 18:11:50.477424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476930    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy"
Nov 02 18:11:50.477424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476941    4456 cpu_manager.go:414] "containerMap: removing container" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy"
Nov 02 18:11:50.477424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.476945    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy"
Nov 02 18:11:50.477424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.477034    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="694c7cd8-25b5-4f5e-95f6-8cd71f0f51f3" containerName="istio-proxy"
Nov 02 18:11:50.477424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.477044    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="0f2089f4-8ec7-45ce-a2df-0bf024b4158f" containerName="discovery"
Nov 02 18:11:50.477424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.477051    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy"
Nov 02 18:11:50.477424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.477056    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ef6443b3-b452-4868-9cde-f762cf522284" containerName="istio-proxy"
Nov 02 18:11:50.477424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.477064    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="778b9a36-6a7b-4251-a91a-999e358d8c5c" containerName="istio-proxy"
Nov 02 18:11:50.477424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.477070    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d99092cc-b126-48db-8762-a4b8d64ad350" containerName="revoke-keys"
Nov 02 18:11:50.477424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.477075    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="73849b3d-673e-41a8-aff1-db9f0a9cce85" containerName="istio-proxy"
Nov 02 18:11:50.477424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.477080    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy"
Nov 02 18:11:50.477424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.477085    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="408afe2d-75a0-4537-b58c-9ab31f161685" containerName="istio-proxy"
Nov 02 18:11:50.477711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.477091    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6aa41ecd-4a93-4b17-8e84-3f3c018c4d21" containerName="istio-proxy"
Nov 02 18:11:50.477711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.477098    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy"
Nov 02 18:11:50.477711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.477103    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f54bd962-e719-41c4-9c28-56460e46d0f6" containerName="revoke-keys"
Nov 02 18:11:50.477711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.477112    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" containerName="payload-processing"
Nov 02 18:11:50.477711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.477117    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5bb882f6-fd8f-46ae-9396-b52498c5d9db" containerName="istio-proxy"
Nov 02 18:11:50.480187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.480169    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"
Nov 02 18:11:50.496858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.496797    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:11:50.496931 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.496921    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:11:50.500642 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.500619    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"]
Nov 02 18:11:50.561100 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.561063    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d258fd-4f61-4a46-94f6-009c3c370e05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58\" (UID: \"55d258fd-4f61-4a46-94f6-009c3c370e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"
Nov 02 18:11:50.561269 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.561236    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jhqr9\" (UniqueName: \"kubernetes.io/projected/55d258fd-4f61-4a46-94f6-009c3c370e05-kube-api-access-jhqr9\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58\" (UID: \"55d258fd-4f61-4a46-94f6-009c3c370e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"
Nov 02 18:11:50.561356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.561338    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d258fd-4f61-4a46-94f6-009c3c370e05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58\" (UID: \"55d258fd-4f61-4a46-94f6-009c3c370e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"
Nov 02 18:11:50.561629 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.561594    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6phtq\" (UniqueName: \"kubernetes.io/projected/c0ff0cbb-4acf-4d63-9579-6aeac659b2ea-kube-api-access-6phtq\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:50.562527 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.562503    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:11:50.562604 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.562557    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:11:50.628878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.628840    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:11:50.629004 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.628908    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:11:50.662689 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.662646    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jhqr9\" (UniqueName: \"kubernetes.io/projected/55d258fd-4f61-4a46-94f6-009c3c370e05-kube-api-access-jhqr9\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58\" (UID: \"55d258fd-4f61-4a46-94f6-009c3c370e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"
Nov 02 18:11:50.662883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.662862    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d258fd-4f61-4a46-94f6-009c3c370e05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58\" (UID: \"55d258fd-4f61-4a46-94f6-009c3c370e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"
Nov 02 18:11:50.663147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.663126    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d258fd-4f61-4a46-94f6-009c3c370e05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58\" (UID: \"55d258fd-4f61-4a46-94f6-009c3c370e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"
Nov 02 18:11:50.663769 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.663698    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d258fd-4f61-4a46-94f6-009c3c370e05-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58\" (UID: \"55d258fd-4f61-4a46-94f6-009c3c370e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"
Nov 02 18:11:50.665435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.665407    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d258fd-4f61-4a46-94f6-009c3c370e05-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58\" (UID: \"55d258fd-4f61-4a46-94f6-009c3c370e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"
Nov 02 18:11:50.687938 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.687916    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jhqr9\" (UniqueName: \"kubernetes.io/projected/55d258fd-4f61-4a46-94f6-009c3c370e05-kube-api-access-jhqr9\") pod \"maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58\" (UID: \"55d258fd-4f61-4a46-94f6-009c3c370e05\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"
Nov 02 18:11:50.710945 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.710908    4456 patch_prober.go:29] interesting pod/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:11:50.711328 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.710997    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:11:50.794535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:50.794453    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"
Nov 02 18:11:51.110859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.107724    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"]
Nov 02 18:11:51.110859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.108715    4456 cpu_manager.go:414] "containerMap: removing container" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy"
Nov 02 18:11:51.110859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.108734    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy"
Nov 02 18:11:51.110859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.108975    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="cb09170f-1b12-4a26-8e5b-c5e2c2964a4d" containerName="istio-proxy"
Nov 02 18:11:51.110859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.108997    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="999df778-cd9b-4b41-852d-4661a7326483" containerName="istio-proxy"
Nov 02 18:11:51.120478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.120454    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"
Nov 02 18:11:51.131089 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.131063    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"]
Nov 02 18:11:51.241353 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.241317    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g" event={"ID":"c0ff0cbb-4acf-4d63-9579-6aeac659b2ea","Type":"ContainerDied","Data":"c31100af0a57f3e463e2a59f7b244d7da03e818a6765fc5a0fa944ebc056a2e8"}
Nov 02 18:11:51.241544 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.241363    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"
Nov 02 18:11:51.241544 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.241382    4456 scope.go:122] "RemoveContainer" containerID="ab3b1839843faf727063beb731cd4336c7b223cdec36d36195c04bd9689f1da3"
Nov 02 18:11:51.256635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.256606    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"]
Nov 02 18:11:51.266915 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.266892    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"
Nov 02 18:11:51.283003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.282563    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ec64b7-c3f5-4493-8657-07fe497a12aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr\" (UID: \"d8ec64b7-c3f5-4493-8657-07fe497a12aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"
Nov 02 18:11:51.283003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.282607    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ec64b7-c3f5-4493-8657-07fe497a12aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr\" (UID: \"d8ec64b7-c3f5-4493-8657-07fe497a12aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"
Nov 02 18:11:51.283003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.282636    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2qwrd\" (UniqueName: \"kubernetes.io/projected/d8ec64b7-c3f5-4493-8657-07fe497a12aa-kube-api-access-2qwrd\") pod \"maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr\" (UID: \"d8ec64b7-c3f5-4493-8657-07fe497a12aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"
Nov 02 18:11:51.283976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.283925    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"]
Nov 02 18:11:51.290765 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.290718    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"]
Nov 02 18:11:51.309104 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.309070    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"]
Nov 02 18:11:51.317369 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.317342    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-inf-b-8a209f-79647b8f8c-zhq8g"]
Nov 02 18:11:51.323674 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.323644    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"]
Nov 02 18:11:51.384245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.384212    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h8x5n\" (UniqueName: \"kubernetes.io/projected/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-kube-api-access-h8x5n\") pod \"maas-api-revoke-keys-e2e-inf-a-a29999-vmshb\" (UID: \"2c58a7cf-d0de-4692-bc36-7d02d346eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"
Nov 02 18:11:51.384382 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.384346    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a29999-vmshb\" (UID: \"2c58a7cf-d0de-4692-bc36-7d02d346eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"
Nov 02 18:11:51.384482 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.384465    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a29999-vmshb\" (UID: \"2c58a7cf-d0de-4692-bc36-7d02d346eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"
Nov 02 18:11:51.384573 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.384559    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ec64b7-c3f5-4493-8657-07fe497a12aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr\" (UID: \"d8ec64b7-c3f5-4493-8657-07fe497a12aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"
Nov 02 18:11:51.384632 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.384586    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ec64b7-c3f5-4493-8657-07fe497a12aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr\" (UID: \"d8ec64b7-c3f5-4493-8657-07fe497a12aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"
Nov 02 18:11:51.384632 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.384613    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2qwrd\" (UniqueName: \"kubernetes.io/projected/d8ec64b7-c3f5-4493-8657-07fe497a12aa-kube-api-access-2qwrd\") pod \"maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr\" (UID: \"d8ec64b7-c3f5-4493-8657-07fe497a12aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"
Nov 02 18:11:51.386182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.385786    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ec64b7-c3f5-4493-8657-07fe497a12aa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr\" (UID: \"d8ec64b7-c3f5-4493-8657-07fe497a12aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"
Nov 02 18:11:51.390923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.390877    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ec64b7-c3f5-4493-8657-07fe497a12aa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr\" (UID: \"d8ec64b7-c3f5-4493-8657-07fe497a12aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"
Nov 02 18:11:51.425532 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.425499    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2qwrd\" (UniqueName: \"kubernetes.io/projected/d8ec64b7-c3f5-4493-8657-07fe497a12aa-kube-api-access-2qwrd\") pod \"maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr\" (UID: \"d8ec64b7-c3f5-4493-8657-07fe497a12aa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"
Nov 02 18:11:51.457841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.456471    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"
Nov 02 18:11:51.457841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.457554    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:51.488101 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.486640    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a29999-vmshb\" (UID: \"2c58a7cf-d0de-4692-bc36-7d02d346eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"
Nov 02 18:11:51.488101 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.486899    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h8x5n\" (UniqueName: \"kubernetes.io/projected/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-kube-api-access-h8x5n\") pod \"maas-api-revoke-keys-e2e-inf-a-a29999-vmshb\" (UID: \"2c58a7cf-d0de-4692-bc36-7d02d346eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"
Nov 02 18:11:51.488101 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.487022    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a29999-vmshb\" (UID: \"2c58a7cf-d0de-4692-bc36-7d02d346eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"
Nov 02 18:11:51.488565 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.488536    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a29999-vmshb\" (UID: \"2c58a7cf-d0de-4692-bc36-7d02d346eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"
Nov 02 18:11:51.492437 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.492417    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-a29999-vmshb\" (UID: \"2c58a7cf-d0de-4692-bc36-7d02d346eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"
Nov 02 18:11:51.515138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.515099    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h8x5n\" (UniqueName: \"kubernetes.io/projected/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-kube-api-access-h8x5n\") pod \"maas-api-revoke-keys-e2e-inf-a-a29999-vmshb\" (UID: \"2c58a7cf-d0de-4692-bc36-7d02d346eb2e\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"
Nov 02 18:11:51.595324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.594763    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"
Nov 02 18:11:51.621607 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.621555    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c0ff0cbb-4acf-4d63-9579-6aeac659b2ea" path="/var/lib/kubelet/pods/c0ff0cbb-4acf-4d63-9579-6aeac659b2ea/volumes"
Nov 02 18:11:51.625908 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.625879    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:51.842334 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.834310    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:51.910439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.910409    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"]
Nov 02 18:11:51.936998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.925306    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"]
Nov 02 18:11:51.943223 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.940106    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"]
Nov 02 18:11:51.943223 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:51.940474    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"
Nov 02 18:11:52.013062 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.013019    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tttcf\" (UniqueName: \"kubernetes.io/projected/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-kube-api-access-tttcf\") pod \"maas-api-revoke-keys-e2e-inf-a-872354-ppd4j\" (UID: \"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"
Nov 02 18:11:52.013209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.013190    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-872354-ppd4j\" (UID: \"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"
Nov 02 18:11:52.013546 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.013511    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-872354-ppd4j\" (UID: \"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"
Nov 02 18:11:52.117354 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.116309    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-872354-ppd4j\" (UID: \"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"
Nov 02 18:11:52.117354 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.116407    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tttcf\" (UniqueName: \"kubernetes.io/projected/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-kube-api-access-tttcf\") pod \"maas-api-revoke-keys-e2e-inf-a-872354-ppd4j\" (UID: \"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"
Nov 02 18:11:52.117354 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.116540    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-872354-ppd4j\" (UID: \"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"
Nov 02 18:11:52.118104 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.118077    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-872354-ppd4j\" (UID: \"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"
Nov 02 18:11:52.131951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.131847    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-872354-ppd4j\" (UID: \"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"
Nov 02 18:11:52.148635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.148536    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tttcf\" (UniqueName: \"kubernetes.io/projected/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-kube-api-access-tttcf\") pod \"maas-api-revoke-keys-e2e-inf-a-872354-ppd4j\" (UID: \"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"
Nov 02 18:11:52.206659 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.206624    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"]
Nov 02 18:11:52.422448 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.422416    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:52.428237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.428124    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58" event={"ID":"55d258fd-4f61-4a46-94f6-009c3c370e05","Type":"ContainerStarted","Data":"0312f401b957bf64103e1e2ec02100ed3dd682d64ce32741d12e521889093217"}
Nov 02 18:11:52.428237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.428188    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58" event={"ID":"55d258fd-4f61-4a46-94f6-009c3c370e05","Type":"ContainerStarted","Data":"4d5d9ff1bc212d10861bf630963d8a71cabb00197ad92e18384d8bbf8a35b38f"}
Nov 02 18:11:52.442807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.440637    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr" event={"ID":"d8ec64b7-c3f5-4493-8657-07fe497a12aa","Type":"ContainerStarted","Data":"c660b5af7306d9c04dd3b0e08f6d6bca442d8402b6363871efddf11a602035a6"}
Nov 02 18:11:52.461883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.447386    4456 generic.go:356] "Generic (PLEG): container finished" podID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerID="7d77b8d6127469706b8666ac8056de7ddc308b1c2fb37e9a35ec932c0e735f54" exitCode=0
Nov 02 18:11:52.461883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.447580    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" event={"ID":"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e","Type":"ContainerDied","Data":"7d77b8d6127469706b8666ac8056de7ddc308b1c2fb37e9a35ec932c0e735f54"}
Nov 02 18:11:52.461883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.450642    4456 generic.go:356] "Generic (PLEG): container finished" podID="07f874bb-122b-4794-ab59-736495e5cfdc" containerID="aff8684245e6811977d173dba51890ae99616b8073fb04935c6a2893e1dadaf3" exitCode=0
Nov 02 18:11:52.461883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.450771    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" event={"ID":"07f874bb-122b-4794-ab59-736495e5cfdc","Type":"ContainerDied","Data":"aff8684245e6811977d173dba51890ae99616b8073fb04935c6a2893e1dadaf3"}
Nov 02 18:11:52.461883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.453052    4456 generic.go:356] "Generic (PLEG): container finished" podID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerID="99a6168054233269cb6e8344874ad06a145a87f2aae8004e7910f961ede56963" exitCode=0
Nov 02 18:11:52.461883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.453130    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" event={"ID":"5775b5a6-6407-46f4-84ef-96c708d7934b","Type":"ContainerDied","Data":"99a6168054233269cb6e8344874ad06a145a87f2aae8004e7910f961ede56963"}
Nov 02 18:11:52.461883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.455477    4456 generic.go:356] "Generic (PLEG): container finished" podID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerID="95adba9c28b54d3f0298de1dec2292a9810fd828ad712b762f15dc288bbab452" exitCode=0
Nov 02 18:11:52.462470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.455553    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" event={"ID":"38977edb-8612-48c6-a6c6-1714a50a9e4f","Type":"ContainerDied","Data":"95adba9c28b54d3f0298de1dec2292a9810fd828ad712b762f15dc288bbab452"}
Nov 02 18:11:52.462470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.457787    4456 generic.go:356] "Generic (PLEG): container finished" podID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerID="730da757c8ffa0ec7438177532c8a6ce99b4b9d761e88b8e9e1bb45179e1eca6" exitCode=0
Nov 02 18:11:52.462470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.457877    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" event={"ID":"81fd82a8-2bbf-43c8-962c-f8384287dc1f","Type":"ContainerDied","Data":"730da757c8ffa0ec7438177532c8a6ce99b4b9d761e88b8e9e1bb45179e1eca6"}
Nov 02 18:11:52.462470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.460250    4456 generic.go:356] "Generic (PLEG): container finished" podID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerID="7ea8d9afddf0f6e4f043e8c148239bb0b12e1d9cf9c2338a03073a647dc99771" exitCode=0
Nov 02 18:11:52.462470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.460407    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" event={"ID":"806cc8ce-f608-4975-94dd-5edade11fc8c","Type":"ContainerDied","Data":"7ea8d9afddf0f6e4f043e8c148239bb0b12e1d9cf9c2338a03073a647dc99771"}
Nov 02 18:11:52.464083 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.463929    4456 generic.go:356] "Generic (PLEG): container finished" podID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerID="60addef8887b57cc6daa92f4b0dbb1eb9ed3aa5f9e9c2ecde7a1e071117ccff8" exitCode=0
Nov 02 18:11:52.464083 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.464017    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" event={"ID":"280a3e6c-c540-4c66-b963-c678ddca6e40","Type":"ContainerDied","Data":"60addef8887b57cc6daa92f4b0dbb1eb9ed3aa5f9e9c2ecde7a1e071117ccff8"}
Nov 02 18:11:52.467258 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.467091    4456 generic.go:356] "Generic (PLEG): container finished" podID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerID="6f672ff50b415c9f5d877d39fc43cbb4be0f8cf802d9f572dcc1a950321d5ed3" exitCode=0
Nov 02 18:11:52.467258 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.467181    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" event={"ID":"8cba3353-fa51-41d3-acb4-a3cfbaae1daa","Type":"ContainerDied","Data":"6f672ff50b415c9f5d877d39fc43cbb4be0f8cf802d9f572dcc1a950321d5ed3"}
Nov 02 18:11:52.470567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.470255    4456 generic.go:356] "Generic (PLEG): container finished" podID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerID="97a86bee14d24f04dff0a2c6c82d05fab51870580f831e892bca987e63a5078c" exitCode=0
Nov 02 18:11:52.470567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.470341    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" event={"ID":"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0","Type":"ContainerDied","Data":"97a86bee14d24f04dff0a2c6c82d05fab51870580f831e892bca987e63a5078c"}
Nov 02 18:11:52.470567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.470407    4456 scope.go:122] "RemoveContainer" containerID="a1b54e6df78108d7c52226ef6abfb9dcf23c4f7a7ff0b1f2e044b8ac4afdddec"
Nov 02 18:11:52.508959 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.508889    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58" podStartSLOduration=2.508868212 podStartE2EDuration="2.508868212s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:52.492194314 +0000 UTC m=+2528.983257480" watchObservedRunningTime="2026-11-02 18:11:52.508868212 +0000 UTC m=+2528.999931374"
Nov 02 18:11:52.533496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.533468    4456 generic.go:356] "Generic (PLEG): container finished" podID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerID="4cb6844084f3272f4de73238806060cd7601c7e4e7b1a56e862c1e414d1e1d67" exitCode=0
Nov 02 18:11:52.533782 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.533762    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" event={"ID":"dbd34da0-9b90-458d-9581-ed9caf3daa11","Type":"ContainerDied","Data":"4cb6844084f3272f4de73238806060cd7601c7e4e7b1a56e862c1e414d1e1d67"}
Nov 02 18:11:52.551837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.544210    4456 generic.go:356] "Generic (PLEG): container finished" podID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerID="34675a9bb3e0d78cbe38155c7ebf2cd25bd2500003af0cd5b590e6ab4a83d61a" exitCode=0
Nov 02 18:11:52.551837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.544272    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" event={"ID":"5b0ff069-2a17-489e-ab3d-6d5461bffbf4","Type":"ContainerDied","Data":"34675a9bb3e0d78cbe38155c7ebf2cd25bd2500003af0cd5b590e6ab4a83d61a"}
Nov 02 18:11:52.580185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.577523    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:52.618913 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.618752    4456 scope.go:122] "RemoveContainer" containerID="a602b0afbc3a3601e8c878664a9fa511e8e88746ceb9ca1f1a2a019ef2cff058"
Nov 02 18:11:52.620265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.620243    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"
Nov 02 18:11:52.624622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.623990    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:11:52.629320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.629300    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 18:11:52.687555 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.684190    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:11:52.729027 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.728998    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:11:52.735592 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.735570    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:11:52.739855 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.739797    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:52.745184 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.745158    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-credential-socket\") pod \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") "
Nov 02 18:11:52.745328 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.745313    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-podinfo\") pod \"806cc8ce-f608-4975-94dd-5edade11fc8c\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") "
Nov 02 18:11:52.745447 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.745433    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-envoy\") pod \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") "
Nov 02 18:11:52.745552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.745540    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-credential-socket\") pod \"dbd34da0-9b90-458d-9581-ed9caf3daa11\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") "
Nov 02 18:11:52.745677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.745664    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-kube-api-access-b75mv\" (UniqueName: \"kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-kube-api-access-b75mv\") pod \"dbd34da0-9b90-458d-9581-ed9caf3daa11\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") "
Nov 02 18:11:52.745808 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.745797    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-data\") pod \"dbd34da0-9b90-458d-9581-ed9caf3daa11\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") "
Nov 02 18:11:52.745948 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.745931    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-podinfo\") pod \"dbd34da0-9b90-458d-9581-ed9caf3daa11\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") "
Nov 02 18:11:52.746070 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.746054    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-workload-socket\") pod \"806cc8ce-f608-4975-94dd-5edade11fc8c\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") "
Nov 02 18:11:52.746185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.746169    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-envoy\") pod \"806cc8ce-f608-4975-94dd-5edade11fc8c\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") "
Nov 02 18:11:52.746315 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.746290    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-workload-socket\") pod \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") "
Nov 02 18:11:52.746442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.746428    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dbd34da0-9b90-458d-9581-ed9caf3daa11-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd34da0-9b90-458d-9581-ed9caf3daa11-istiod-ca-cert\") pod \"dbd34da0-9b90-458d-9581-ed9caf3daa11\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") "
Nov 02 18:11:52.746557 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.746543    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-data\") pod \"806cc8ce-f608-4975-94dd-5edade11fc8c\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") "
Nov 02 18:11:52.746739 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.746723    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-workload-socket\") pod \"dbd34da0-9b90-458d-9581-ed9caf3daa11\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") "
Nov 02 18:11:52.746878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.746863    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806cc8ce-f608-4975-94dd-5edade11fc8c-kube-api-access-g9h5l\" (UniqueName: \"kubernetes.io/projected/806cc8ce-f608-4975-94dd-5edade11fc8c-kube-api-access-g9h5l\") pod \"806cc8ce-f608-4975-94dd-5edade11fc8c\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") "
Nov 02 18:11:52.747287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.747268    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-token\" (UniqueName: \"kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-token\") pod \"dbd34da0-9b90-458d-9581-ed9caf3daa11\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") "
Nov 02 18:11:52.749220 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.747419    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-token\" (UniqueName: \"kubernetes.io/projected/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-token\") pod \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") "
Nov 02 18:11:52.749359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.749345    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-podinfo\") pod \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") "
Nov 02 18:11:52.749460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.749447    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-workload-certs\") pod \"806cc8ce-f608-4975-94dd-5edade11fc8c\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") "
Nov 02 18:11:52.749582 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.749536    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-envoy\") pod \"dbd34da0-9b90-458d-9581-ed9caf3daa11\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") "
Nov 02 18:11:52.749728 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.749710    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-workload-certs\") pod \"dbd34da0-9b90-458d-9581-ed9caf3daa11\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") "
Nov 02 18:11:52.750113 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.750086    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-workload-certs" pod "dbd34da0-9b90-458d-9581-ed9caf3daa11" (UID: "dbd34da0-9b90-458d-9581-ed9caf3daa11"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.750455 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.750436    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-credential-socket" pod "5b0ff069-2a17-489e-ab3d-6d5461bffbf4" (UID: "5b0ff069-2a17-489e-ab3d-6d5461bffbf4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.750921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.750899    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/806cc8ce-f608-4975-94dd-5edade11fc8c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/806cc8ce-f608-4975-94dd-5edade11fc8c-istiod-ca-cert\") pod \"806cc8ce-f608-4975-94dd-5edade11fc8c\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") "
Nov 02 18:11:52.770435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.768772    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-data\") pod \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") "
Nov 02 18:11:52.770435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.768887    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-workload-certs\") pod \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") "
Nov 02 18:11:52.770435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.768917    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istiod-ca-cert\") pod \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") "
Nov 02 18:11:52.770435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.768944    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-token\" (UniqueName: \"kubernetes.io/projected/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-token\") pod \"806cc8ce-f608-4975-94dd-5edade11fc8c\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") "
Nov 02 18:11:52.770435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.768997    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-credential-socket\") pod \"806cc8ce-f608-4975-94dd-5edade11fc8c\" (UID: \"806cc8ce-f608-4975-94dd-5edade11fc8c\") "
Nov 02 18:11:52.770744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.769026    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-kube-api-access-9ms8q\" (UniqueName: \"kubernetes.io/projected/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-kube-api-access-9ms8q\") pod \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\" (UID: \"5b0ff069-2a17-489e-ab3d-6d5461bffbf4\") "
Nov 02 18:11:52.770744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.769998    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.770744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.770035    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.770744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.751694    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/806cc8ce-f608-4975-94dd-5edade11fc8c-istiod-ca-cert" pod "806cc8ce-f608-4975-94dd-5edade11fc8c" (UID: "806cc8ce-f608-4975-94dd-5edade11fc8c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:52.770744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.770218    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-workload-socket" pod "5b0ff069-2a17-489e-ab3d-6d5461bffbf4" (UID: "5b0ff069-2a17-489e-ab3d-6d5461bffbf4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.770744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.753929    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-data" pod "dbd34da0-9b90-458d-9581-ed9caf3daa11" (UID: "dbd34da0-9b90-458d-9581-ed9caf3daa11"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.771010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.753968    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-credential-socket" pod "dbd34da0-9b90-458d-9581-ed9caf3daa11" (UID: "dbd34da0-9b90-458d-9581-ed9caf3daa11"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.771010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.754157    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-workload-certs" pod "806cc8ce-f608-4975-94dd-5edade11fc8c" (UID: "806cc8ce-f608-4975-94dd-5edade11fc8c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.771010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.754696    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-workload-socket" pod "dbd34da0-9b90-458d-9581-ed9caf3daa11" (UID: "dbd34da0-9b90-458d-9581-ed9caf3daa11"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.771010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.752233    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-data" pod "806cc8ce-f608-4975-94dd-5edade11fc8c" (UID: "806cc8ce-f608-4975-94dd-5edade11fc8c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.771986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.771224    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-data" pod "5b0ff069-2a17-489e-ab3d-6d5461bffbf4" (UID: "5b0ff069-2a17-489e-ab3d-6d5461bffbf4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.771986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.771442    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-workload-certs" pod "5b0ff069-2a17-489e-ab3d-6d5461bffbf4" (UID: "5b0ff069-2a17-489e-ab3d-6d5461bffbf4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.771986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.771710    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istiod-ca-cert" pod "5b0ff069-2a17-489e-ab3d-6d5461bffbf4" (UID: "5b0ff069-2a17-489e-ab3d-6d5461bffbf4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:52.774204 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.774184    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-workload-socket" pod "806cc8ce-f608-4975-94dd-5edade11fc8c" (UID: "806cc8ce-f608-4975-94dd-5edade11fc8c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.776997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.776970    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dbd34da0-9b90-458d-9581-ed9caf3daa11-istiod-ca-cert" pod "dbd34da0-9b90-458d-9581-ed9caf3daa11" (UID: "dbd34da0-9b90-458d-9581-ed9caf3daa11"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:52.786482 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.786347    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-credential-socket" pod "806cc8ce-f608-4975-94dd-5edade11fc8c" (UID: "806cc8ce-f608-4975-94dd-5edade11fc8c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.879591 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.879547    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-kube-api-access-wx6r2\" (UniqueName: \"kubernetes.io/projected/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-kube-api-access-wx6r2\") pod \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") "
Nov 02 18:11:52.889783 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.887645    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-envoy\") pod \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") "
Nov 02 18:11:52.889992 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.889975    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istiod-ca-cert\") pod \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") "
Nov 02 18:11:52.890269 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.890235    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-podinfo\") pod \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") "
Nov 02 18:11:52.890485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.890429    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-workload-certs\") pod \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") "
Nov 02 18:11:52.890604 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.890591    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-data\") pod \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") "
Nov 02 18:11:52.890708 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.890684    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-workload-certs" pod "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" (UID: "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.890855 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.890838    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-credential-socket\") pod \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") "
Nov 02 18:11:52.891013 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.890989    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-data" pod "8cba3353-fa51-41d3-acb4-a3cfbaae1daa" (UID: "8cba3353-fa51-41d3-acb4-a3cfbaae1daa"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.891463 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.891028    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istiod-ca-cert" pod "8cba3353-fa51-41d3-acb4-a3cfbaae1daa" (UID: "8cba3353-fa51-41d3-acb4-a3cfbaae1daa"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:52.891463 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.891176    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-credential-socket" pod "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" (UID: "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.892405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.892302    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806cc8ce-f608-4975-94dd-5edade11fc8c-kube-api-access-g9h5l" pod "806cc8ce-f608-4975-94dd-5edade11fc8c" (UID: "806cc8ce-f608-4975-94dd-5edade11fc8c"). InnerVolumeSpecName "kube-api-access-g9h5l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:52.893033 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.890998    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-podinfo\") pod \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") "
Nov 02 18:11:52.897658 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.897525    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istiod-ca-cert\") pod \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") "
Nov 02 18:11:52.897658 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.896778    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-kube-api-access-b75mv" pod "dbd34da0-9b90-458d-9581-ed9caf3daa11" (UID: "dbd34da0-9b90-458d-9581-ed9caf3daa11"). InnerVolumeSpecName "kube-api-access-b75mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:52.897658 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.897064    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-podinfo" pod "806cc8ce-f608-4975-94dd-5edade11fc8c" (UID: "806cc8ce-f608-4975-94dd-5edade11fc8c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:52.897834 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.897735    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-kube-api-access-b75mv\" (UniqueName: \"kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-kube-api-access-b75mv\") pod \"dbd34da0-9b90-458d-9581-ed9caf3daa11\" (UID: \"dbd34da0-9b90-458d-9581-ed9caf3daa11\") "
Nov 02 18:11:52.897890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.897807    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-token\" (UniqueName: \"kubernetes.io/projected/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-token\") pod \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") "
Nov 02 18:11:52.900029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.897946    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-workload-certs\") pod \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") "
Nov 02 18:11:52.900029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.898000    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-data\") pod \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") "
Nov 02 18:11:52.900029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.898032    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-credential-socket\") pod \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") "
Nov 02 18:11:52.900029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.898071    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-token\" (UniqueName: \"kubernetes.io/projected/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-token\") pod \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") "
Nov 02 18:11:52.900029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.898119    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-workload-socket\") pod \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") "
Nov 02 18:11:52.900316 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.898193    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-envoy\") pod \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\" (UID: \"8cba3353-fa51-41d3-acb4-a3cfbaae1daa\") "
Nov 02 18:11:52.900316 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.898230    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-kube-api-access-g75j4\" (UniqueName: \"kubernetes.io/projected/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-kube-api-access-g75j4\") pod \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") "
Nov 02 18:11:52.900316 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.898263    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-workload-socket\") pod \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\" (UID: \"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e\") "
Nov 02 18:11:52.900316 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899171    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.900316 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899193    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/806cc8ce-f608-4975-94dd-5edade11fc8c-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.900316 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899206    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.900596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899220    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.900596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899232    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.900596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899245    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.900596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899258    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.900596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899271    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.900596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899285    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.900596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899297    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.900596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899311    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.901012 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899324    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.901012 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899339    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.901012 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899352    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.901012 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899366    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/dbd34da0-9b90-458d-9581-ed9caf3daa11-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.901012 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899380    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.901012 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899393    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.901012 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899407    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g9h5l\" (UniqueName: \"kubernetes.io/projected/806cc8ce-f608-4975-94dd-5edade11fc8c-kube-api-access-g9h5l\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.901012 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.899421    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:52.901380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.900766    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-envoy" pod "dbd34da0-9b90-458d-9581-ed9caf3daa11" (UID: "dbd34da0-9b90-458d-9581-ed9caf3daa11"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.901380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.901042    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-envoy" pod "5b0ff069-2a17-489e-ab3d-6d5461bffbf4" (UID: "5b0ff069-2a17-489e-ab3d-6d5461bffbf4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.901928 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:52.901503    4456 empty_dir.go:505] Warning: Unmount skipped because path does not exist: /var/lib/kubelet/pods/dbd34da0-9b90-458d-9581-ed9caf3daa11/volumes/kubernetes.io~projected/kube-api-access-b75mv
Nov 02 18:11:52.901928 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.901359    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-workload-socket" pod "8cba3353-fa51-41d3-acb4-a3cfbaae1daa" (UID: "8cba3353-fa51-41d3-acb4-a3cfbaae1daa"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.901928 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.901557    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-kube-api-access-b75mv" pod "dbd34da0-9b90-458d-9581-ed9caf3daa11" (UID: "dbd34da0-9b90-458d-9581-ed9caf3daa11"). InnerVolumeSpecName "kube-api-access-b75mv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:52.902374 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.902354    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-workload-socket" pod "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" (UID: "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.915500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.904377    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-data" pod "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" (UID: "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.915500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.904402    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-workload-certs" pod "8cba3353-fa51-41d3-acb4-a3cfbaae1daa" (UID: "8cba3353-fa51-41d3-acb4-a3cfbaae1daa"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.915500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.904603    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-credential-socket" pod "8cba3353-fa51-41d3-acb4-a3cfbaae1daa" (UID: "8cba3353-fa51-41d3-acb4-a3cfbaae1daa"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.915500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.905224    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-podinfo" pod "5b0ff069-2a17-489e-ab3d-6d5461bffbf4" (UID: "5b0ff069-2a17-489e-ab3d-6d5461bffbf4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:52.915500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.912277    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-podinfo" pod "dbd34da0-9b90-458d-9581-ed9caf3daa11" (UID: "dbd34da0-9b90-458d-9581-ed9caf3daa11"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:52.915792 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.914380    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istiod-ca-cert" pod "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" (UID: "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:52.929729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.920944    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-token" pod "806cc8ce-f608-4975-94dd-5edade11fc8c" (UID: "806cc8ce-f608-4975-94dd-5edade11fc8c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:52.929729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.924592    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-token" pod "dbd34da0-9b90-458d-9581-ed9caf3daa11" (UID: "dbd34da0-9b90-458d-9581-ed9caf3daa11"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:52.929729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.926030    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-token" pod "5b0ff069-2a17-489e-ab3d-6d5461bffbf4" (UID: "5b0ff069-2a17-489e-ab3d-6d5461bffbf4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:52.929729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.928001    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-envoy" pod "806cc8ce-f608-4975-94dd-5edade11fc8c" (UID: "806cc8ce-f608-4975-94dd-5edade11fc8c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.929729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.928109    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-kube-api-access-wx6r2" pod "8cba3353-fa51-41d3-acb4-a3cfbaae1daa" (UID: "8cba3353-fa51-41d3-acb4-a3cfbaae1daa"). InnerVolumeSpecName "kube-api-access-wx6r2". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:52.931234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.929220    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-envoy" pod "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" (UID: "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.931234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.929586    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-kube-api-access-9ms8q" pod "5b0ff069-2a17-489e-ab3d-6d5461bffbf4" (UID: "5b0ff069-2a17-489e-ab3d-6d5461bffbf4"). InnerVolumeSpecName "kube-api-access-9ms8q". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:52.931234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.931085    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-podinfo" pod "8cba3353-fa51-41d3-acb4-a3cfbaae1daa" (UID: "8cba3353-fa51-41d3-acb4-a3cfbaae1daa"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:52.931234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.931150    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-envoy" pod "8cba3353-fa51-41d3-acb4-a3cfbaae1daa" (UID: "8cba3353-fa51-41d3-acb4-a3cfbaae1daa"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:52.931234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.931179    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-kube-api-access-g75j4" pod "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" (UID: "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e"). InnerVolumeSpecName "kube-api-access-g75j4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:52.934464 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.934435    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-token" pod "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" (UID: "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:52.987645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.985596    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-podinfo" pod "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" (UID: "5e54b0f0-5c3c-4441-a2a6-b452b9715e2e"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:52.987645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:52.986102    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-token" pod "8cba3353-fa51-41d3-acb4-a3cfbaae1daa" (UID: "8cba3353-fa51-41d3-acb4-a3cfbaae1daa"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:53.001383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.000942    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.001383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.000967    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g75j4\" (UniqueName: \"kubernetes.io/projected/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-kube-api-access-g75j4\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.001383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.000980    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.001383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.000993    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wx6r2\" (UniqueName: \"kubernetes.io/projected/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-kube-api-access-wx6r2\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.001383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001005    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.001383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001017    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.001383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001028    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.001383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001039    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.001803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001050    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.001803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001061    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.001803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001073    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.001803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001084    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.001803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001095    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9ms8q\" (UniqueName: \"kubernetes.io/projected/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-kube-api-access-9ms8q\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.001803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001107    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.001803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001119    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5b0ff069-2a17-489e-ab3d-6d5461bffbf4-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.001803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001130    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-b75mv\" (UniqueName: \"kubernetes.io/projected/dbd34da0-9b90-458d-9581-ed9caf3daa11-kube-api-access-b75mv\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.002172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001142    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.002172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001155    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/dbd34da0-9b90-458d-9581-ed9caf3daa11-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.002172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001167    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/806cc8ce-f608-4975-94dd-5edade11fc8c-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.002172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001178    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.002172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001189    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.002172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001201    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.002172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001213    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.002172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.001227    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/8cba3353-fa51-41d3-acb4-a3cfbaae1daa-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.017934 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.017903    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 18:11:53.022619 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.020941    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:53.043782 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.043705    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 18:11:53.045651 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.045586    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 18:11:53.110328 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.102700    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-workload-certs\") pod \"5775b5a6-6407-46f4-84ef-96c708d7934b\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") "
Nov 02 18:11:53.110328 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.102855    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-credential-socket\") pod \"5775b5a6-6407-46f4-84ef-96c708d7934b\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") "
Nov 02 18:11:53.110328 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.103450    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-podinfo\") pod \"5775b5a6-6407-46f4-84ef-96c708d7934b\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") "
Nov 02 18:11:53.110328 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.103580    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-workload-socket\") pod \"5775b5a6-6407-46f4-84ef-96c708d7934b\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") "
Nov 02 18:11:53.110328 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.104064    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-credential-socket" pod "5775b5a6-6407-46f4-84ef-96c708d7934b" (UID: "5775b5a6-6407-46f4-84ef-96c708d7934b"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.120734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.104661    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-workload-socket" pod "5775b5a6-6407-46f4-84ef-96c708d7934b" (UID: "5775b5a6-6407-46f4-84ef-96c708d7934b"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.120734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.104961    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-token\" (UniqueName: \"kubernetes.io/projected/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-token\") pod \"5775b5a6-6407-46f4-84ef-96c708d7934b\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") "
Nov 02 18:11:53.120734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.105048    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-envoy\") pod \"5775b5a6-6407-46f4-84ef-96c708d7934b\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") "
Nov 02 18:11:53.120734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.105267    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-data\") pod \"5775b5a6-6407-46f4-84ef-96c708d7934b\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") "
Nov 02 18:11:53.120734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.105346    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/5775b5a6-6407-46f4-84ef-96c708d7934b-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5775b5a6-6407-46f4-84ef-96c708d7934b-istiod-ca-cert\") pod \"5775b5a6-6407-46f4-84ef-96c708d7934b\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") "
Nov 02 18:11:53.121265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.105494    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5775b5a6-6407-46f4-84ef-96c708d7934b-kube-api-access-nqv2l\" (UniqueName: \"kubernetes.io/projected/5775b5a6-6407-46f4-84ef-96c708d7934b-kube-api-access-nqv2l\") pod \"5775b5a6-6407-46f4-84ef-96c708d7934b\" (UID: \"5775b5a6-6407-46f4-84ef-96c708d7934b\") "
Nov 02 18:11:53.121265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.106145    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-workload-certs" pod "5775b5a6-6407-46f4-84ef-96c708d7934b" (UID: "5775b5a6-6407-46f4-84ef-96c708d7934b"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.121265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.110240    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.121265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.110265    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.121265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.110301    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.121265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.119941    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/5775b5a6-6407-46f4-84ef-96c708d7934b-istiod-ca-cert" pod "5775b5a6-6407-46f4-84ef-96c708d7934b" (UID: "5775b5a6-6407-46f4-84ef-96c708d7934b"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:53.121676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.120024    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-data" pod "5775b5a6-6407-46f4-84ef-96c708d7934b" (UID: "5775b5a6-6407-46f4-84ef-96c708d7934b"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.121676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.120452    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-token" pod "5775b5a6-6407-46f4-84ef-96c708d7934b" (UID: "5775b5a6-6407-46f4-84ef-96c708d7934b"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:53.121676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.121447    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-podinfo" pod "5775b5a6-6407-46f4-84ef-96c708d7934b" (UID: "5775b5a6-6407-46f4-84ef-96c708d7934b"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:53.124011 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.123974    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5775b5a6-6407-46f4-84ef-96c708d7934b-kube-api-access-nqv2l" pod "5775b5a6-6407-46f4-84ef-96c708d7934b" (UID: "5775b5a6-6407-46f4-84ef-96c708d7934b"). InnerVolumeSpecName "kube-api-access-nqv2l". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:53.191939 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.177934    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-envoy" pod "5775b5a6-6407-46f4-84ef-96c708d7934b" (UID: "5775b5a6-6407-46f4-84ef-96c708d7934b"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.214054 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.211286    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istiod-ca-cert\") pod \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") "
Nov 02 18:11:53.214054 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.211347    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-token\" (UniqueName: \"kubernetes.io/projected/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-token\") pod \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") "
Nov 02 18:11:53.214054 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.211387    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f874bb-122b-4794-ab59-736495e5cfdc-istio-token\" (UniqueName: \"kubernetes.io/projected/07f874bb-122b-4794-ab59-736495e5cfdc-istio-token\") pod \"07f874bb-122b-4794-ab59-736495e5cfdc\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") "
Nov 02 18:11:53.214054 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.211466    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-data\") pod \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") "
Nov 02 18:11:53.214054 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.211525    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-workload-certs\") pod \"07f874bb-122b-4794-ab59-736495e5cfdc\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") "
Nov 02 18:11:53.214405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.211554    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/07f874bb-122b-4794-ab59-736495e5cfdc-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f874bb-122b-4794-ab59-736495e5cfdc-istio-podinfo\") pod \"07f874bb-122b-4794-ab59-736495e5cfdc\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") "
Nov 02 18:11:53.214405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.211586    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-istio-data\") pod \"07f874bb-122b-4794-ab59-736495e5cfdc\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") "
Nov 02 18:11:53.214405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.211838    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-workload-socket\") pod \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") "
Nov 02 18:11:53.214405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.211873    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-podinfo\") pod \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") "
Nov 02 18:11:53.214405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.211904    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-envoy\") pod \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") "
Nov 02 18:11:53.214624 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.211975    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-workload-socket\") pod \"07f874bb-122b-4794-ab59-736495e5cfdc\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") "
Nov 02 18:11:53.214624 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.212037    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/07f874bb-122b-4794-ab59-736495e5cfdc-kube-api-access-245hq\" (UniqueName: \"kubernetes.io/projected/07f874bb-122b-4794-ab59-736495e5cfdc-kube-api-access-245hq\") pod \"07f874bb-122b-4794-ab59-736495e5cfdc\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") "
Nov 02 18:11:53.214624 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.212064    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-workload-certs\") pod \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") "
Nov 02 18:11:53.214624 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.212098    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/81fd82a8-2bbf-43c8-962c-f8384287dc1f-kube-api-access-mfs8x\" (UniqueName: \"kubernetes.io/projected/81fd82a8-2bbf-43c8-962c-f8384287dc1f-kube-api-access-mfs8x\") pod \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") "
Nov 02 18:11:53.214624 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.212169    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-credential-socket\") pod \"07f874bb-122b-4794-ab59-736495e5cfdc\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") "
Nov 02 18:11:53.214850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.212267    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-istio-envoy\") pod \"07f874bb-122b-4794-ab59-736495e5cfdc\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") "
Nov 02 18:11:53.214850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.212320    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/07f874bb-122b-4794-ab59-736495e5cfdc-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f874bb-122b-4794-ab59-736495e5cfdc-istiod-ca-cert\") pod \"07f874bb-122b-4794-ab59-736495e5cfdc\" (UID: \"07f874bb-122b-4794-ab59-736495e5cfdc\") "
Nov 02 18:11:53.214850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.212352    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-credential-socket\") pod \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\" (UID: \"81fd82a8-2bbf-43c8-962c-f8384287dc1f\") "
Nov 02 18:11:53.214850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.213041    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.214850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.213064    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.214850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.213078    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.215121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.213091    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/5775b5a6-6407-46f4-84ef-96c708d7934b-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.215121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.213104    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nqv2l\" (UniqueName: \"kubernetes.io/projected/5775b5a6-6407-46f4-84ef-96c708d7934b-kube-api-access-nqv2l\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.215121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.213124    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/5775b5a6-6407-46f4-84ef-96c708d7934b-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.215121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.213385    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-credential-socket" pod "81fd82a8-2bbf-43c8-962c-f8384287dc1f" (UID: "81fd82a8-2bbf-43c8-962c-f8384287dc1f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.215121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.213703    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istiod-ca-cert" pod "81fd82a8-2bbf-43c8-962c-f8384287dc1f" (UID: "81fd82a8-2bbf-43c8-962c-f8384287dc1f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:53.217623 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.216614    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-istio-data" pod "07f874bb-122b-4794-ab59-736495e5cfdc" (UID: "07f874bb-122b-4794-ab59-736495e5cfdc"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.217623 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.216835    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-data" pod "81fd82a8-2bbf-43c8-962c-f8384287dc1f" (UID: "81fd82a8-2bbf-43c8-962c-f8384287dc1f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.219163 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.218736    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-workload-socket" pod "81fd82a8-2bbf-43c8-962c-f8384287dc1f" (UID: "81fd82a8-2bbf-43c8-962c-f8384287dc1f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.221431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.221401    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-workload-certs" pod "07f874bb-122b-4794-ab59-736495e5cfdc" (UID: "07f874bb-122b-4794-ab59-736495e5cfdc"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.223757 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.221644    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-workload-socket" pod "07f874bb-122b-4794-ab59-736495e5cfdc" (UID: "07f874bb-122b-4794-ab59-736495e5cfdc"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.223757 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.223637    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/07f874bb-122b-4794-ab59-736495e5cfdc-istiod-ca-cert" pod "07f874bb-122b-4794-ab59-736495e5cfdc" (UID: "07f874bb-122b-4794-ab59-736495e5cfdc"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:53.223757 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.223669    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-credential-socket" pod "07f874bb-122b-4794-ab59-736495e5cfdc" (UID: "07f874bb-122b-4794-ab59-736495e5cfdc"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.226245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.226206    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-workload-certs" pod "81fd82a8-2bbf-43c8-962c-f8384287dc1f" (UID: "81fd82a8-2bbf-43c8-962c-f8384287dc1f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.230241 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.229748    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:11:53.233656 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.233526    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 18:11:53.234863 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.234838    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-envoy" pod "81fd82a8-2bbf-43c8-962c-f8384287dc1f" (UID: "81fd82a8-2bbf-43c8-962c-f8384287dc1f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.234946 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.234911    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/07f874bb-122b-4794-ab59-736495e5cfdc-istio-podinfo" pod "07f874bb-122b-4794-ab59-736495e5cfdc" (UID: "07f874bb-122b-4794-ab59-736495e5cfdc"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:53.235723 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.235691    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-token" pod "81fd82a8-2bbf-43c8-962c-f8384287dc1f" (UID: "81fd82a8-2bbf-43c8-962c-f8384287dc1f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:53.237163 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.237131    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f874bb-122b-4794-ab59-736495e5cfdc-istio-token" pod "07f874bb-122b-4794-ab59-736495e5cfdc" (UID: "07f874bb-122b-4794-ab59-736495e5cfdc"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:53.237527 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.237501    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/81fd82a8-2bbf-43c8-962c-f8384287dc1f-kube-api-access-mfs8x" pod "81fd82a8-2bbf-43c8-962c-f8384287dc1f" (UID: "81fd82a8-2bbf-43c8-962c-f8384287dc1f"). InnerVolumeSpecName "kube-api-access-mfs8x". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:53.239578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.239556    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-podinfo" pod "81fd82a8-2bbf-43c8-962c-f8384287dc1f" (UID: "81fd82a8-2bbf-43c8-962c-f8384287dc1f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:53.239757 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.239723    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/07f874bb-122b-4794-ab59-736495e5cfdc-kube-api-access-245hq" pod "07f874bb-122b-4794-ab59-736495e5cfdc" (UID: "07f874bb-122b-4794-ab59-736495e5cfdc"). InnerVolumeSpecName "kube-api-access-245hq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:53.239849 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.239779    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-istio-envoy" pod "07f874bb-122b-4794-ab59-736495e5cfdc" (UID: "07f874bb-122b-4794-ab59-736495e5cfdc"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.289080 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.289027    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 18:11:53.295875 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.295485    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 18:11:53.318947 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.318894    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-envoy\") pod \"280a3e6c-c540-4c66-b963-c678ddca6e40\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") "
Nov 02 18:11:53.319760 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.319703    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/38977edb-8612-48c6-a6c6-1714a50a9e4f-istiod-ca-cert" pod "38977edb-8612-48c6-a6c6-1714a50a9e4f" (UID: "38977edb-8612-48c6-a6c6-1714a50a9e4f"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:53.319966 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.319945    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/38977edb-8612-48c6-a6c6-1714a50a9e4f-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38977edb-8612-48c6-a6c6-1714a50a9e4f-istiod-ca-cert\") pod \"38977edb-8612-48c6-a6c6-1714a50a9e4f\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") "
Nov 02 18:11:53.320143 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.320119    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-token\" (UniqueName: \"kubernetes.io/projected/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-token\") pod \"280a3e6c-c540-4c66-b963-c678ddca6e40\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") "
Nov 02 18:11:53.322594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.322142    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-podinfo\") pod \"280a3e6c-c540-4c66-b963-c678ddca6e40\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") "
Nov 02 18:11:53.322594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.321556    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"]
Nov 02 18:11:53.322594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.322209    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38977edb-8612-48c6-a6c6-1714a50a9e4f-kube-api-access-xn778\" (UniqueName: \"kubernetes.io/projected/38977edb-8612-48c6-a6c6-1714a50a9e4f-kube-api-access-xn778\") pod \"38977edb-8612-48c6-a6c6-1714a50a9e4f\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") "
Nov 02 18:11:53.322594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.322272    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-workload-certs\") pod \"280a3e6c-c540-4c66-b963-c678ddca6e40\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") "
Nov 02 18:11:53.322594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.322324    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-workload-socket\") pod \"280a3e6c-c540-4c66-b963-c678ddca6e40\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") "
Nov 02 18:11:53.322594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.322405    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-envoy\") pod \"38977edb-8612-48c6-a6c6-1714a50a9e4f\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") "
Nov 02 18:11:53.323064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.322447    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/280a3e6c-c540-4c66-b963-c678ddca6e40-kube-api-access-5f64s\" (UniqueName: \"kubernetes.io/projected/280a3e6c-c540-4c66-b963-c678ddca6e40-kube-api-access-5f64s\") pod \"280a3e6c-c540-4c66-b963-c678ddca6e40\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") "
Nov 02 18:11:53.323064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.322480    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-credential-socket\") pod \"280a3e6c-c540-4c66-b963-c678ddca6e40\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") "
Nov 02 18:11:53.323064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.322560    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-credential-socket\") pod \"38977edb-8612-48c6-a6c6-1714a50a9e4f\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") "
Nov 02 18:11:53.323064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.322593    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-podinfo\") pod \"38977edb-8612-48c6-a6c6-1714a50a9e4f\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") "
Nov 02 18:11:53.323064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.322636    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-data\") pod \"38977edb-8612-48c6-a6c6-1714a50a9e4f\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") "
Nov 02 18:11:53.325980 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.322665    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-data\") pod \"280a3e6c-c540-4c66-b963-c678ddca6e40\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") "
Nov 02 18:11:53.325980 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.324871    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-workload-certs\") pod \"38977edb-8612-48c6-a6c6-1714a50a9e4f\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") "
Nov 02 18:11:53.325980 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.324959    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/280a3e6c-c540-4c66-b963-c678ddca6e40-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280a3e6c-c540-4c66-b963-c678ddca6e40-istiod-ca-cert\") pod \"280a3e6c-c540-4c66-b963-c678ddca6e40\" (UID: \"280a3e6c-c540-4c66-b963-c678ddca6e40\") "
Nov 02 18:11:53.325980 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325010    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-workload-socket\") pod \"38977edb-8612-48c6-a6c6-1714a50a9e4f\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") "
Nov 02 18:11:53.325980 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325078    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-token\" (UniqueName: \"kubernetes.io/projected/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-token\") pod \"38977edb-8612-48c6-a6c6-1714a50a9e4f\" (UID: \"38977edb-8612-48c6-a6c6-1714a50a9e4f\") "
Nov 02 18:11:53.326234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325778    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/07f874bb-122b-4794-ab59-736495e5cfdc-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.326234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325798    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.326234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325829    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.326234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325843    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.326234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325856    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/07f874bb-122b-4794-ab59-736495e5cfdc-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.326234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325868    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.326234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325883    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.326234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325895    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/07f874bb-122b-4794-ab59-736495e5cfdc-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.326597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325907    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.326597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325919    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.326597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325931    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.326597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325943    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.326597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325956    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.326597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325968    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-245hq\" (UniqueName: \"kubernetes.io/projected/07f874bb-122b-4794-ab59-736495e5cfdc-kube-api-access-245hq\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.326597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325981    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/81fd82a8-2bbf-43c8-962c-f8384287dc1f-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.326597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.325995    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mfs8x\" (UniqueName: \"kubernetes.io/projected/81fd82a8-2bbf-43c8-962c-f8384287dc1f-kube-api-access-mfs8x\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.327192 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.326009    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.327192 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.326023    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/38977edb-8612-48c6-a6c6-1714a50a9e4f-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.327192 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.326037    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/07f874bb-122b-4794-ab59-736495e5cfdc-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.327485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.327424    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-podinfo" pod "280a3e6c-c540-4c66-b963-c678ddca6e40" (UID: "280a3e6c-c540-4c66-b963-c678ddca6e40"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:53.332052 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.330215    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-data" pod "38977edb-8612-48c6-a6c6-1714a50a9e4f" (UID: "38977edb-8612-48c6-a6c6-1714a50a9e4f"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.332052 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.330635    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-workload-certs" pod "38977edb-8612-48c6-a6c6-1714a50a9e4f" (UID: "38977edb-8612-48c6-a6c6-1714a50a9e4f"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.332052 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.331090    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/280a3e6c-c540-4c66-b963-c678ddca6e40-istiod-ca-cert" pod "280a3e6c-c540-4c66-b963-c678ddca6e40" (UID: "280a3e6c-c540-4c66-b963-c678ddca6e40"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:53.332052 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.331783    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-credential-socket" pod "38977edb-8612-48c6-a6c6-1714a50a9e4f" (UID: "38977edb-8612-48c6-a6c6-1714a50a9e4f"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.332052 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.332025    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-workload-socket" pod "38977edb-8612-48c6-a6c6-1714a50a9e4f" (UID: "38977edb-8612-48c6-a6c6-1714a50a9e4f"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.332585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.332401    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-credential-socket" pod "280a3e6c-c540-4c66-b963-c678ddca6e40" (UID: "280a3e6c-c540-4c66-b963-c678ddca6e40"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.332585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.332462    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-workload-certs" pod "280a3e6c-c540-4c66-b963-c678ddca6e40" (UID: "280a3e6c-c540-4c66-b963-c678ddca6e40"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.332585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.332500    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-workload-socket" pod "280a3e6c-c540-4c66-b963-c678ddca6e40" (UID: "280a3e6c-c540-4c66-b963-c678ddca6e40"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.332585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.332538    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-data" pod "280a3e6c-c540-4c66-b963-c678ddca6e40" (UID: "280a3e6c-c540-4c66-b963-c678ddca6e40"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.332585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.332561    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-envoy" pod "280a3e6c-c540-4c66-b963-c678ddca6e40" (UID: "280a3e6c-c540-4c66-b963-c678ddca6e40"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.339594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.339561    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-token" pod "280a3e6c-c540-4c66-b963-c678ddca6e40" (UID: "280a3e6c-c540-4c66-b963-c678ddca6e40"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:53.341421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.341388    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-envoy" pod "38977edb-8612-48c6-a6c6-1714a50a9e4f" (UID: "38977edb-8612-48c6-a6c6-1714a50a9e4f"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.341992 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.341954    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-podinfo" pod "38977edb-8612-48c6-a6c6-1714a50a9e4f" (UID: "38977edb-8612-48c6-a6c6-1714a50a9e4f"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:53.366186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.360923    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38977edb-8612-48c6-a6c6-1714a50a9e4f-kube-api-access-xn778" pod "38977edb-8612-48c6-a6c6-1714a50a9e4f" (UID: "38977edb-8612-48c6-a6c6-1714a50a9e4f"). InnerVolumeSpecName "kube-api-access-xn778". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:53.366186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.362890    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/280a3e6c-c540-4c66-b963-c678ddca6e40-kube-api-access-5f64s" pod "280a3e6c-c540-4c66-b963-c678ddca6e40" (UID: "280a3e6c-c540-4c66-b963-c678ddca6e40"). InnerVolumeSpecName "kube-api-access-5f64s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:53.366767 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.366496    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-token" pod "38977edb-8612-48c6-a6c6-1714a50a9e4f" (UID: "38977edb-8612-48c6-a6c6-1714a50a9e4f"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:53.371476 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:53.371427    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod71f8e0ab_cd0d_42ad_a56b_f8c22c7208cc.slice/crio-5fa680781ccf2aabe1b1b25efb418dd360f8dddd0f3bec5482c22e420d654be0 WatchSource:0}: Error finding container 5fa680781ccf2aabe1b1b25efb418dd360f8dddd0f3bec5482c22e420d654be0: Status 404 returned error can't find the container with id 5fa680781ccf2aabe1b1b25efb418dd360f8dddd0f3bec5482c22e420d654be0
Nov 02 18:11:53.430550 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.429530    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-podinfo\") pod \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") "
Nov 02 18:11:53.430550 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.429604    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-data\") pod \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") "
Nov 02 18:11:53.430550 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.429683    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-workload-certs\") pod \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") "
Nov 02 18:11:53.430550 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.429732    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istiod-ca-cert\") pod \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") "
Nov 02 18:11:53.430550 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.429782    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-workload-socket\") pod \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") "
Nov 02 18:11:53.432205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.432159    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-workload-certs" pod "e5167f1d-3718-4c96-bf19-1cdc1ab73207" (UID: "e5167f1d-3718-4c96-bf19-1cdc1ab73207"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.435058 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.434533    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istiod-ca-cert" pod "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" (UID: "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:53.438611 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.438559    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-podinfo" pod "e5167f1d-3718-4c96-bf19-1cdc1ab73207" (UID: "e5167f1d-3718-4c96-bf19-1cdc1ab73207"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:53.442909 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.440072    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-workload-socket" pod "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" (UID: "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.447090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.445571    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istiod-ca-cert\") pod \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") "
Nov 02 18:11:53.447090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.445734    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-podinfo\") pod \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") "
Nov 02 18:11:53.447090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.445772    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-token\" (UniqueName: \"kubernetes.io/projected/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-token\") pod \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") "
Nov 02 18:11:53.447090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.445806    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5167f1d-3718-4c96-bf19-1cdc1ab73207-kube-api-access-tsncm\" (UniqueName: \"kubernetes.io/projected/e5167f1d-3718-4c96-bf19-1cdc1ab73207-kube-api-access-tsncm\") pod \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") "
Nov 02 18:11:53.447090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.445870    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-data\") pod \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") "
Nov 02 18:11:53.447382 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.445939    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-workload-socket\") pod \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") "
Nov 02 18:11:53.447382 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.445970    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-token\" (UniqueName: \"kubernetes.io/projected/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-token\") pod \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") "
Nov 02 18:11:53.447382 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.446012    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-credential-socket\") pod \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") "
Nov 02 18:11:53.447382 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.446044    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-envoy\") pod \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") "
Nov 02 18:11:53.447382 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.446082    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-envoy\") pod \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") "
Nov 02 18:11:53.449040 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.446140    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-kube-api-access-cp2xr\" (UniqueName: \"kubernetes.io/projected/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-kube-api-access-cp2xr\") pod \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") "
Nov 02 18:11:53.449040 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.446189    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-credential-socket\") pod \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\" (UID: \"e5167f1d-3718-4c96-bf19-1cdc1ab73207\") "
Nov 02 18:11:53.449040 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.446231    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-workload-certs\") pod \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\" (UID: \"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0\") "
Nov 02 18:11:53.449040 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.446532    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-data" pod "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" (UID: "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.449040 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.446766    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istiod-ca-cert" pod "e5167f1d-3718-4c96-bf19-1cdc1ab73207" (UID: "e5167f1d-3718-4c96-bf19-1cdc1ab73207"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:53.449291 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447360    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.449291 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447384    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.449291 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447406    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.449291 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447422    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.449291 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447435    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.449291 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447448    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.449291 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447467    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.449291 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447513    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xn778\" (UniqueName: \"kubernetes.io/projected/38977edb-8612-48c6-a6c6-1714a50a9e4f-kube-api-access-xn778\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.449645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447528    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.449645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447543    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.449645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447573    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-data" pod "e5167f1d-3718-4c96-bf19-1cdc1ab73207" (UID: "e5167f1d-3718-4c96-bf19-1cdc1ab73207"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.449645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447635    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.449645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447652    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.449645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447665    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.449645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447680    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.450102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447700    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.450102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447715    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5f64s\" (UniqueName: \"kubernetes.io/projected/280a3e6c-c540-4c66-b963-c678ddca6e40-kube-api-access-5f64s\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.450102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447730    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.450102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447750    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.450102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447766    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.450102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447779    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.450102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447792    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/280a3e6c-c540-4c66-b963-c678ddca6e40-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.450102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447828    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/38977edb-8612-48c6-a6c6-1714a50a9e4f-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.450491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447843    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/280a3e6c-c540-4c66-b963-c678ddca6e40-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.450491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.447847    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-workload-socket" pod "e5167f1d-3718-4c96-bf19-1cdc1ab73207" (UID: "e5167f1d-3718-4c96-bf19-1cdc1ab73207"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.450491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.448118    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-workload-certs" pod "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" (UID: "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.450491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.448330    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-credential-socket" pod "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" (UID: "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.450491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.448645    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-credential-socket" pod "e5167f1d-3718-4c96-bf19-1cdc1ab73207" (UID: "e5167f1d-3718-4c96-bf19-1cdc1ab73207"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.450729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.449894    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5167f1d-3718-4c96-bf19-1cdc1ab73207-kube-api-access-tsncm" pod "e5167f1d-3718-4c96-bf19-1cdc1ab73207" (UID: "e5167f1d-3718-4c96-bf19-1cdc1ab73207"). InnerVolumeSpecName "kube-api-access-tsncm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:53.458558 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.458180    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-podinfo" pod "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" (UID: "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:11:53.458558 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.458217    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-envoy" pod "e5167f1d-3718-4c96-bf19-1cdc1ab73207" (UID: "e5167f1d-3718-4c96-bf19-1cdc1ab73207"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.458558 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.458495    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-token" pod "e5167f1d-3718-4c96-bf19-1cdc1ab73207" (UID: "e5167f1d-3718-4c96-bf19-1cdc1ab73207"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:53.458558 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.458516    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-kube-api-access-cp2xr" pod "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" (UID: "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0"). InnerVolumeSpecName "kube-api-access-cp2xr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:53.460242 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.460218    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-token" pod "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" (UID: "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:53.460360 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.460335    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-envoy" pod "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" (UID: "ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:11:53.542514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.542474    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"]
Nov 02 18:11:53.543495 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543467    4456 cpu_manager.go:414] "containerMap: removing container" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy"
Nov 02 18:11:53.543495 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543495    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy"
Nov 02 18:11:53.543615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543517    4456 cpu_manager.go:414] "containerMap: removing container" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy"
Nov 02 18:11:53.543615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543525    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy"
Nov 02 18:11:53.543615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543543    4456 cpu_manager.go:414] "containerMap: removing container" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy"
Nov 02 18:11:53.543615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543553    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy"
Nov 02 18:11:53.543615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543571    4456 cpu_manager.go:414] "containerMap: removing container" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy"
Nov 02 18:11:53.543615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543579    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy"
Nov 02 18:11:53.543615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543601    4456 cpu_manager.go:414] "containerMap: removing container" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy"
Nov 02 18:11:53.543615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543608    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy"
Nov 02 18:11:53.543850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543625    4456 cpu_manager.go:414] "containerMap: removing container" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy"
Nov 02 18:11:53.543850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543633    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy"
Nov 02 18:11:53.543850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543657    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerName="istio-proxy"
Nov 02 18:11:53.543850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543665    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerName="istio-proxy"
Nov 02 18:11:53.543850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543678    4456 cpu_manager.go:414] "containerMap: removing container" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy"
Nov 02 18:11:53.543850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543685    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy"
Nov 02 18:11:53.543850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543696    4456 cpu_manager.go:414] "containerMap: removing container" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy"
Nov 02 18:11:53.543850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543703    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy"
Nov 02 18:11:53.543850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543715    4456 cpu_manager.go:414] "containerMap: removing container" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy"
Nov 02 18:11:53.543850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543722    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy"
Nov 02 18:11:53.543850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543744    4456 cpu_manager.go:414] "containerMap: removing container" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy"
Nov 02 18:11:53.543850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543752    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy"
Nov 02 18:11:53.544146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543770    4456 cpu_manager.go:414] "containerMap: removing container" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy"
Nov 02 18:11:53.544146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543777    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy"
Nov 02 18:11:53.544146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543786    4456 cpu_manager.go:414] "containerMap: removing container" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy"
Nov 02 18:11:53.544146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543792    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy"
Nov 02 18:11:53.544146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543807    4456 cpu_manager.go:414] "containerMap: removing container" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy"
Nov 02 18:11:53.544146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.543832    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy"
Nov 02 18:11:53.544146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.544006    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" containerName="istio-proxy"
Nov 02 18:11:53.544146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.544024    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" containerName="istio-proxy"
Nov 02 18:11:53.544146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.544036    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" containerName="istio-proxy"
Nov 02 18:11:53.544146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.544048    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" containerName="istio-proxy"
Nov 02 18:11:53.544146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.544060    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" containerName="istio-proxy"
Nov 02 18:11:53.544146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.544071    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" containerName="istio-proxy"
Nov 02 18:11:53.544431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.544085    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerName="istio-proxy"
Nov 02 18:11:53.544431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.544095    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy"
Nov 02 18:11:53.544431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.544102    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" containerName="istio-proxy"
Nov 02 18:11:53.544431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.544110    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" containerName="istio-proxy"
Nov 02 18:11:53.544431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.544119    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" containerName="istio-proxy"
Nov 02 18:11:53.544431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.544131    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy"
Nov 02 18:11:53.548731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.548571    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.548731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.548605    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.548731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.548631    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tsncm\" (UniqueName: \"kubernetes.io/projected/e5167f1d-3718-4c96-bf19-1cdc1ab73207-kube-api-access-tsncm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.548731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.548642    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.548731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.548656    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.548731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.548664    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.548731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.548672    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.548731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.548680    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.549073 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.548687    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.549073 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.548695    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cp2xr\" (UniqueName: \"kubernetes.io/projected/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-kube-api-access-cp2xr\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.549073 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.548704    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e5167f1d-3718-4c96-bf19-1cdc1ab73207-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.549073 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.548712    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:53.549560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.549531    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"
Nov 02 18:11:53.577793 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.577762    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths" event={"ID":"5775b5a6-6407-46f4-84ef-96c708d7934b","Type":"ContainerDied","Data":"724b2f34a5f83d96e3ed0b3f0e724fa1b26929da5345368ee4fe9957a36e154f"}
Nov 02 18:11:53.577941 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.577802    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"
Nov 02 18:11:53.578337 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.578315    4456 scope.go:122] "RemoveContainer" containerID="99a6168054233269cb6e8344874ad06a145a87f2aae8004e7910f961ede56963"
Nov 02 18:11:53.617840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.584914    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"]
Nov 02 18:11:53.617840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.594456    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:53.629966 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.629923    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"
Nov 02 18:11:53.631167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.631144    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"
Nov 02 18:11:53.640113 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.638796    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"
Nov 02 18:11:53.641766 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.641736    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx" event={"ID":"38977edb-8612-48c6-a6c6-1714a50a9e4f","Type":"ContainerDied","Data":"74225bd3ac6241676ac73b0acba4384e47d913bbf76c3afc6d0e20827d4fe06b"}
Nov 02 18:11:53.658204 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.658173    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx" event={"ID":"81fd82a8-2bbf-43c8-962c-f8384287dc1f","Type":"ContainerDied","Data":"b86b3ef9420d3f4216826fe71374370eb03bb3f93f859161af397d62b99ab672"}
Nov 02 18:11:53.658466 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.658444    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46" event={"ID":"806cc8ce-f608-4975-94dd-5edade11fc8c","Type":"ContainerDied","Data":"55c5ffef20baf90e86b8e6c4709e524db068d7d1b2bad7ab617d4d741b856797"}
Nov 02 18:11:53.659148 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.659122    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a230f129-4b8a-4544-9e35-053804d1edd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr\" (UID: \"a230f129-4b8a-4544-9e35-053804d1edd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"
Nov 02 18:11:53.659278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.659239    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p7fjh\" (UniqueName: \"kubernetes.io/projected/a230f129-4b8a-4544-9e35-053804d1edd7-kube-api-access-p7fjh\") pod \"maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr\" (UID: \"a230f129-4b8a-4544-9e35-053804d1edd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"
Nov 02 18:11:53.659378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.659345    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a230f129-4b8a-4544-9e35-053804d1edd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr\" (UID: \"a230f129-4b8a-4544-9e35-053804d1edd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"
Nov 02 18:11:53.659889 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.659866    4456 scope.go:122] "RemoveContainer" containerID="95adba9c28b54d3f0298de1dec2292a9810fd828ad712b762f15dc288bbab452"
Nov 02 18:11:53.660774 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.660753    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck" event={"ID":"280a3e6c-c540-4c66-b963-c678ddca6e40","Type":"ContainerDied","Data":"3d9b8adee15845131a45f38a576a6f1283f6e2b9826d6dac59798f22488855ec"}
Nov 02 18:11:53.661332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.660799    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"
Nov 02 18:11:53.664886 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.664745    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"
Nov 02 18:11:53.665043 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.665006    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d" event={"ID":"8cba3353-fa51-41d3-acb4-a3cfbaae1daa","Type":"ContainerDied","Data":"8d9e2ce44f4c31d7512dd74da1f7a331b19098c6711368e074e5c1729d3cf186"}
Nov 02 18:11:53.667519 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.667500    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc" event={"ID":"ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0","Type":"ContainerDied","Data":"2731fa237307e8ca02049b0c93ee8edf1340ab68b0f85380c97410539d6ab10d"}
Nov 02 18:11:53.667605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.667594    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"
Nov 02 18:11:53.670322 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.670291    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"
Nov 02 18:11:53.671382 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.671361    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm" event={"ID":"dbd34da0-9b90-458d-9581-ed9caf3daa11","Type":"ContainerDied","Data":"3e83f28511fc73768824a8bafbe547677dbeba837ccadd9f8b6eeb2e836fa5e8"}
Nov 02 18:11:53.673116 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.673042    4456 generic.go:356] "Generic (PLEG): container finished" podID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" containerID="aaaa12bbfe6411dd90db578524e8a9702da102f37573e9977362c806a70e72b3" exitCode=0
Nov 02 18:11:53.673177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.673110    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"
Nov 02 18:11:53.673209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.673092    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" event={"ID":"e5167f1d-3718-4c96-bf19-1cdc1ab73207","Type":"ContainerDied","Data":"aaaa12bbfe6411dd90db578524e8a9702da102f37573e9977362c806a70e72b3"}
Nov 02 18:11:53.673238 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.673219    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f" event={"ID":"e5167f1d-3718-4c96-bf19-1cdc1ab73207","Type":"ContainerDied","Data":"2385f09846039394e104b7d8241a2008a6cfba0db6f192dd2c1616316880d45f"}
Nov 02 18:11:53.675910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.675892    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"
Nov 02 18:11:53.675991 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.675910    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw" event={"ID":"5b0ff069-2a17-489e-ab3d-6d5461bffbf4","Type":"ContainerDied","Data":"bc071adec7a494208311ed9826625b1df3e9551774073ebf93a334bf665d7cb5"}
Nov 02 18:11:53.679689 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.679664    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb" event={"ID":"2c58a7cf-d0de-4692-bc36-7d02d346eb2e","Type":"ContainerStarted","Data":"9867f9993570e8f274c2e9b7b82271760e52385bdbb74f70bc0aa8a29f039774"}
Nov 02 18:11:53.679752 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.679708    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb" event={"ID":"2c58a7cf-d0de-4692-bc36-7d02d346eb2e","Type":"ContainerStarted","Data":"2a4420ece27d37555913f1620cb5c1626a524a45cf2daf2659ca5f748800d9ab"}
Nov 02 18:11:53.682210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.681993    4456 generic.go:356] "Generic (PLEG): container finished" podID="55d258fd-4f61-4a46-94f6-009c3c370e05" containerID="0312f401b957bf64103e1e2ec02100ed3dd682d64ce32741d12e521889093217" exitCode=0
Nov 02 18:11:53.682210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.682066    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58" event={"ID":"55d258fd-4f61-4a46-94f6-009c3c370e05","Type":"ContainerDied","Data":"0312f401b957bf64103e1e2ec02100ed3dd682d64ce32741d12e521889093217"}
Nov 02 18:11:53.716659 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.716383    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr" event={"ID":"d8ec64b7-c3f5-4493-8657-07fe497a12aa","Type":"ContainerStarted","Data":"8edb03650dca6b1935c892249b69bc92c07709952f03b11a0207dba795beaa1c"}
Nov 02 18:11:53.718976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.718952    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"
Nov 02 18:11:53.719252 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.718963    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq" event={"ID":"5e54b0f0-5c3c-4441-a2a6-b452b9715e2e","Type":"ContainerDied","Data":"2300f223a1d1ec0b2987b8462270cde83cca39409c0ffbb2273e03ea9a9f7274"}
Nov 02 18:11:53.726344 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.722958    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j" event={"ID":"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc","Type":"ContainerStarted","Data":"5fa680781ccf2aabe1b1b25efb418dd360f8dddd0f3bec5482c22e420d654be0"}
Nov 02 18:11:53.726344 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.724946    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2" event={"ID":"07f874bb-122b-4794-ab59-736495e5cfdc","Type":"ContainerDied","Data":"d35cd6bf175b5a6b7c0ef33fdc1bc03a436a9ed8ef8a2f695ebb1d2d633e9c6e"}
Nov 02 18:11:53.726344 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.725052    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"
Nov 02 18:11:53.751679 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.751657    4456 scope.go:122] "RemoveContainer" containerID="730da757c8ffa0ec7438177532c8a6ce99b4b9d761e88b8e9e1bb45179e1eca6"
Nov 02 18:11:53.761663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.761075    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a230f129-4b8a-4544-9e35-053804d1edd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr\" (UID: \"a230f129-4b8a-4544-9e35-053804d1edd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"
Nov 02 18:11:53.761663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.761243    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p7fjh\" (UniqueName: \"kubernetes.io/projected/a230f129-4b8a-4544-9e35-053804d1edd7-kube-api-access-p7fjh\") pod \"maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr\" (UID: \"a230f129-4b8a-4544-9e35-053804d1edd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"
Nov 02 18:11:53.761663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.761317    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a230f129-4b8a-4544-9e35-053804d1edd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr\" (UID: \"a230f129-4b8a-4544-9e35-053804d1edd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"
Nov 02 18:11:53.762688 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.762665    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a230f129-4b8a-4544-9e35-053804d1edd7-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr\" (UID: \"a230f129-4b8a-4544-9e35-053804d1edd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"
Nov 02 18:11:53.764690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.764665    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a230f129-4b8a-4544-9e35-053804d1edd7-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr\" (UID: \"a230f129-4b8a-4544-9e35-053804d1edd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"
Nov 02 18:11:53.832234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.832202    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p7fjh\" (UniqueName: \"kubernetes.io/projected/a230f129-4b8a-4544-9e35-053804d1edd7-kube-api-access-p7fjh\") pod \"maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr\" (UID: \"a230f129-4b8a-4544-9e35-053804d1edd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"
Nov 02 18:11:53.878913 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.878882    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"
Nov 02 18:11:53.908018 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.904757    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:53.948744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.941157    4456 scope.go:122] "RemoveContainer" containerID="7ea8d9afddf0f6e4f043e8c148239bb0b12e1d9cf9c2338a03073a647dc99771"
Nov 02 18:11:53.964634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.964604    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"]
Nov 02 18:11:53.967414 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.967344    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" containerName="istio-proxy"
Nov 02 18:11:53.967568 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.967552    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" containerName="istio-proxy"
Nov 02 18:11:53.974047 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.974023    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"]
Nov 02 18:11:53.974643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:53.974620    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"
Nov 02 18:11:54.024630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.024608    4456 scope.go:122] "RemoveContainer" containerID="60addef8887b57cc6daa92f4b0dbb1eb9ed3aa5f9e9c2ecde7a1e071117ccff8"
Nov 02 18:11:54.033209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.033189    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w5-863b87-openshift-default-8f57fd975-ngths"]
Nov 02 18:11:54.047788 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.046755    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"]
Nov 02 18:11:54.125114 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.125084    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ef929e-6a26-44e0-a4d6-f376244672a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q\" (UID: \"30ef929e-6a26-44e0-a4d6-f376244672a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"
Nov 02 18:11:54.125249 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.125156    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ef929e-6a26-44e0-a4d6-f376244672a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q\" (UID: \"30ef929e-6a26-44e0-a4d6-f376244672a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"
Nov 02 18:11:54.125249 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.125193    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8pcdd\" (UniqueName: \"kubernetes.io/projected/30ef929e-6a26-44e0-a4d6-f376244672a8-kube-api-access-8pcdd\") pod \"maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q\" (UID: \"30ef929e-6a26-44e0-a4d6-f376244672a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"
Nov 02 18:11:54.131653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.131630    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"]
Nov 02 18:11:54.139867 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.139846    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-7xlkx"]
Nov 02 18:11:54.158377 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.158355    4456 scope.go:122] "RemoveContainer" containerID="6f672ff50b415c9f5d877d39fc43cbb4be0f8cf802d9f572dcc1a950321d5ed3"
Nov 02 18:11:54.164286 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.164214    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb" podStartSLOduration=3.164197439 podStartE2EDuration="3.164197439s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:54.161578602 +0000 UTC m=+2530.652641780" watchObservedRunningTime="2026-11-02 18:11:54.164197439 +0000 UTC m=+2530.655260603"
Nov 02 18:11:54.179512 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.179490    4456 scope.go:122] "RemoveContainer" containerID="97a86bee14d24f04dff0a2c6c82d05fab51870580f831e892bca987e63a5078c"
Nov 02 18:11:54.214862 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.210521    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr" podStartSLOduration=4.210499776 podStartE2EDuration="4.210499776s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:50 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:54.209578044 +0000 UTC m=+2530.700641230" watchObservedRunningTime="2026-11-02 18:11:54.210499776 +0000 UTC m=+2530.701562938"
Nov 02 18:11:54.226538 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.226463    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ef929e-6a26-44e0-a4d6-f376244672a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q\" (UID: \"30ef929e-6a26-44e0-a4d6-f376244672a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"
Nov 02 18:11:54.226773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.226753    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8pcdd\" (UniqueName: \"kubernetes.io/projected/30ef929e-6a26-44e0-a4d6-f376244672a8-kube-api-access-8pcdd\") pod \"maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q\" (UID: \"30ef929e-6a26-44e0-a4d6-f376244672a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"
Nov 02 18:11:54.227108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.227085    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ef929e-6a26-44e0-a4d6-f376244672a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q\" (UID: \"30ef929e-6a26-44e0-a4d6-f376244672a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"
Nov 02 18:11:54.227510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.227481    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ef929e-6a26-44e0-a4d6-f376244672a8-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q\" (UID: \"30ef929e-6a26-44e0-a4d6-f376244672a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"
Nov 02 18:11:54.238014 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.237977    4456 scope.go:122] "RemoveContainer" containerID="4cb6844084f3272f4de73238806060cd7601c7e4e7b1a56e862c1e414d1e1d67"
Nov 02 18:11:54.239377 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.238385    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ef929e-6a26-44e0-a4d6-f376244672a8-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q\" (UID: \"30ef929e-6a26-44e0-a4d6-f376244672a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"
Nov 02 18:11:54.276351 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.276318    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8pcdd\" (UniqueName: \"kubernetes.io/projected/30ef929e-6a26-44e0-a4d6-f376244672a8-kube-api-access-8pcdd\") pod \"maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q\" (UID: \"30ef929e-6a26-44e0-a4d6-f376244672a8\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"
Nov 02 18:11:54.327023 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.327001    4456 scope.go:122] "RemoveContainer" containerID="aaaa12bbfe6411dd90db578524e8a9702da102f37573e9977362c806a70e72b3"
Nov 02 18:11:54.343316 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.343281    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"]
Nov 02 18:11:54.358224 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.358204    4456 scope.go:122] "RemoveContainer" containerID="aaaa12bbfe6411dd90db578524e8a9702da102f37573e9977362c806a70e72b3"
Nov 02 18:11:54.358533 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:54.358506    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"aaaa12bbfe6411dd90db578524e8a9702da102f37573e9977362c806a70e72b3\": container with ID starting with aaaa12bbfe6411dd90db578524e8a9702da102f37573e9977362c806a70e72b3 not found: ID does not exist" containerID="aaaa12bbfe6411dd90db578524e8a9702da102f37573e9977362c806a70e72b3"
Nov 02 18:11:54.358592 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.358546    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"aaaa12bbfe6411dd90db578524e8a9702da102f37573e9977362c806a70e72b3"} err="failed to get container status \"aaaa12bbfe6411dd90db578524e8a9702da102f37573e9977362c806a70e72b3\": rpc error: code = NotFound desc = could not find container \"aaaa12bbfe6411dd90db578524e8a9702da102f37573e9977362c806a70e72b3\": container with ID starting with aaaa12bbfe6411dd90db578524e8a9702da102f37573e9977362c806a70e72b3 not found: ID does not exist"
Nov 02 18:11:54.358592 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.358572    4456 scope.go:122] "RemoveContainer" containerID="34675a9bb3e0d78cbe38155c7ebf2cd25bd2500003af0cd5b590e6ab4a83d61a"
Nov 02 18:11:54.380465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.380432    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"]
Nov 02 18:11:54.398193 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.397627    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-8a209f-openshift-default-869bd8d945-24qrx"]
Nov 02 18:11:54.411788 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.411405    4456 scope.go:122] "RemoveContainer" containerID="7d77b8d6127469706b8666ac8056de7ddc308b1c2fb37e9a35ec932c0e735f54"
Nov 02 18:11:54.428174 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.428150    4456 scope.go:122] "RemoveContainer" containerID="aff8684245e6811977d173dba51890ae99616b8073fb04935c6a2893e1dadaf3"
Nov 02 18:11:54.455539 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.455513    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"
Nov 02 18:11:54.522888 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.522776    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"]
Nov 02 18:11:54.526442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.526400    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w4-2899ab-openshift-default-795b798dff-mcf7f"]
Nov 02 18:11:54.565149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.565114    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"]
Nov 02 18:11:54.570873 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.570840    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-5tsjw"]
Nov 02 18:11:54.608843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.608228    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"]
Nov 02 18:11:54.625503 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.625463    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w0-069fa5-openshift-default-685bb6c8c5-7l4lc"]
Nov 02 18:11:54.666591 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.666561    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"]
Nov 02 18:11:54.672236 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.672204    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-vrvf2"]
Nov 02 18:11:54.736840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.734834    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"]
Nov 02 18:11:54.737780 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.737753    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-fqp46"]
Nov 02 18:11:54.751930 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.751895    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr" event={"ID":"a230f129-4b8a-4544-9e35-053804d1edd7","Type":"ContainerStarted","Data":"881b8d217e29c0cafa90204803f47557a2733f7b68c4b58a1328caad43471411"}
Nov 02 18:11:54.775045 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.774932    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"]
Nov 02 18:11:54.798849 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.792032    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-wnc6d"]
Nov 02 18:11:54.815991 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.815954    4456 generic.go:356] "Generic (PLEG): container finished" podID="2c58a7cf-d0de-4692-bc36-7d02d346eb2e" containerID="9867f9993570e8f274c2e9b7b82271760e52385bdbb74f70bc0aa8a29f039774" exitCode=0
Nov 02 18:11:54.816249 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.816030    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb" event={"ID":"2c58a7cf-d0de-4692-bc36-7d02d346eb2e","Type":"ContainerDied","Data":"9867f9993570e8f274c2e9b7b82271760e52385bdbb74f70bc0aa8a29f039774"}
Nov 02 18:11:54.818051 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.818028    4456 generic.go:356] "Generic (PLEG): container finished" podID="d8ec64b7-c3f5-4493-8657-07fe497a12aa" containerID="8edb03650dca6b1935c892249b69bc92c07709952f03b11a0207dba795beaa1c" exitCode=0
Nov 02 18:11:54.818164 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.818112    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr" event={"ID":"d8ec64b7-c3f5-4493-8657-07fe497a12aa","Type":"ContainerDied","Data":"8edb03650dca6b1935c892249b69bc92c07709952f03b11a0207dba795beaa1c"}
Nov 02 18:11:54.818278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.818260    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"]
Nov 02 18:11:54.821907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.821886    4456 generic.go:356] "Generic (PLEG): container finished" podID="71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc" containerID="f8d3fa499482e951efc1422c6ad8a74b9e54eb1a76cdf63f582ca889d854e091" exitCode=0
Nov 02 18:11:54.822202 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.822181    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j" event={"ID":"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc","Type":"ContainerDied","Data":"f8d3fa499482e951efc1422c6ad8a74b9e54eb1a76cdf63f582ca889d854e091"}
Nov 02 18:11:54.822882 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.822860    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-c7wqm"]
Nov 02 18:11:54.856021 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.855959    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"]
Nov 02 18:11:54.861751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.861716    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-fbbbf8-openshift-default-779c49db9-tmxwq"]
Nov 02 18:11:54.914006 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.913968    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"]
Nov 02 18:11:54.959994 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.959963    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"]
Nov 02 18:11:54.966340 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:54.966317    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5bqck"]
Nov 02 18:11:55.041773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.041744    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"
Nov 02 18:11:55.141129 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.141091    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/55d258fd-4f61-4a46-94f6-009c3c370e05-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d258fd-4f61-4a46-94f6-009c3c370e05-maas-api-openshift-ca\") pod \"55d258fd-4f61-4a46-94f6-009c3c370e05\" (UID: \"55d258fd-4f61-4a46-94f6-009c3c370e05\") "
Nov 02 18:11:55.141259 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.141141    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/55d258fd-4f61-4a46-94f6-009c3c370e05-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d258fd-4f61-4a46-94f6-009c3c370e05-maas-api-xks-ca\") pod \"55d258fd-4f61-4a46-94f6-009c3c370e05\" (UID: \"55d258fd-4f61-4a46-94f6-009c3c370e05\") "
Nov 02 18:11:55.141259 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.141231    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/55d258fd-4f61-4a46-94f6-009c3c370e05-kube-api-access-jhqr9\" (UniqueName: \"kubernetes.io/projected/55d258fd-4f61-4a46-94f6-009c3c370e05-kube-api-access-jhqr9\") pod \"55d258fd-4f61-4a46-94f6-009c3c370e05\" (UID: \"55d258fd-4f61-4a46-94f6-009c3c370e05\") "
Nov 02 18:11:55.142165 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.142135    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/55d258fd-4f61-4a46-94f6-009c3c370e05-maas-api-openshift-ca" pod "55d258fd-4f61-4a46-94f6-009c3c370e05" (UID: "55d258fd-4f61-4a46-94f6-009c3c370e05"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:55.144205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.144176    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/55d258fd-4f61-4a46-94f6-009c3c370e05-kube-api-access-jhqr9" pod "55d258fd-4f61-4a46-94f6-009c3c370e05" (UID: "55d258fd-4f61-4a46-94f6-009c3c370e05"). InnerVolumeSpecName "kube-api-access-jhqr9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:55.144290 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.144244    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/55d258fd-4f61-4a46-94f6-009c3c370e05-maas-api-xks-ca" pod "55d258fd-4f61-4a46-94f6-009c3c370e05" (UID: "55d258fd-4f61-4a46-94f6-009c3c370e05"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:11:55.243388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.243354    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/55d258fd-4f61-4a46-94f6-009c3c370e05-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:55.243388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.243383    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/55d258fd-4f61-4a46-94f6-009c3c370e05-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:55.243388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.243392    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jhqr9\" (UniqueName: \"kubernetes.io/projected/55d258fd-4f61-4a46-94f6-009c3c370e05-kube-api-access-jhqr9\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:55.638792 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.638755    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:55.641067 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.641033    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="07f874bb-122b-4794-ab59-736495e5cfdc" path="/var/lib/kubelet/pods/07f874bb-122b-4794-ab59-736495e5cfdc/volumes"
Nov 02 18:11:55.641890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.641865    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="280a3e6c-c540-4c66-b963-c678ddca6e40" path="/var/lib/kubelet/pods/280a3e6c-c540-4c66-b963-c678ddca6e40/volumes"
Nov 02 18:11:55.642650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.642629    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="38977edb-8612-48c6-a6c6-1714a50a9e4f" path="/var/lib/kubelet/pods/38977edb-8612-48c6-a6c6-1714a50a9e4f/volumes"
Nov 02 18:11:55.643769 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.643744    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5775b5a6-6407-46f4-84ef-96c708d7934b" path="/var/lib/kubelet/pods/5775b5a6-6407-46f4-84ef-96c708d7934b/volumes"
Nov 02 18:11:55.644478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.644456    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5b0ff069-2a17-489e-ab3d-6d5461bffbf4" path="/var/lib/kubelet/pods/5b0ff069-2a17-489e-ab3d-6d5461bffbf4/volumes"
Nov 02 18:11:55.645216 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.645194    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5e54b0f0-5c3c-4441-a2a6-b452b9715e2e" path="/var/lib/kubelet/pods/5e54b0f0-5c3c-4441-a2a6-b452b9715e2e/volumes"
Nov 02 18:11:55.646470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.646446    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="806cc8ce-f608-4975-94dd-5edade11fc8c" path="/var/lib/kubelet/pods/806cc8ce-f608-4975-94dd-5edade11fc8c/volumes"
Nov 02 18:11:55.647158 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.647140    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="81fd82a8-2bbf-43c8-962c-f8384287dc1f" path="/var/lib/kubelet/pods/81fd82a8-2bbf-43c8-962c-f8384287dc1f/volumes"
Nov 02 18:11:55.647716 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.647696    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8cba3353-fa51-41d3-acb4-a3cfbaae1daa" path="/var/lib/kubelet/pods/8cba3353-fa51-41d3-acb4-a3cfbaae1daa/volumes"
Nov 02 18:11:55.648695 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.648675    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0" path="/var/lib/kubelet/pods/ae55ce0a-88e9-4ad4-bfdd-f55dbd9389b0/volumes"
Nov 02 18:11:55.649279 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.649260    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dbd34da0-9b90-458d-9581-ed9caf3daa11" path="/var/lib/kubelet/pods/dbd34da0-9b90-458d-9581-ed9caf3daa11/volumes"
Nov 02 18:11:55.655883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.655857    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5167f1d-3718-4c96-bf19-1cdc1ab73207" path="/var/lib/kubelet/pods/e5167f1d-3718-4c96-bf19-1cdc1ab73207/volumes"
Nov 02 18:11:55.775925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.775898    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-image-registry/image-registry-59767d8795-7vwkt"
Nov 02 18:11:55.860615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.860578    4456 generic.go:356] "Generic (PLEG): container finished" podID="a230f129-4b8a-4544-9e35-053804d1edd7" containerID="fb25944faff8de31645896efa975646aa529d38504de0a61e50895077dfb97f1" exitCode=0
Nov 02 18:11:55.861058 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.861029    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr" event={"ID":"a230f129-4b8a-4544-9e35-053804d1edd7","Type":"ContainerDied","Data":"fb25944faff8de31645896efa975646aa529d38504de0a61e50895077dfb97f1"}
Nov 02 18:11:55.871841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.865949    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58" event={"ID":"55d258fd-4f61-4a46-94f6-009c3c370e05","Type":"ContainerDied","Data":"4d5d9ff1bc212d10861bf630963d8a71cabb00197ad92e18384d8bbf8a35b38f"}
Nov 02 18:11:55.871841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.865995    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4d5d9ff1bc212d10861bf630963d8a71cabb00197ad92e18384d8bbf8a35b38f"
Nov 02 18:11:55.871841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.866067    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"
Nov 02 18:11:55.900839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.888441    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q" event={"ID":"30ef929e-6a26-44e0-a4d6-f376244672a8","Type":"ContainerStarted","Data":"0d54b34fabde9aeebc7f496fae68a4cd7e94840f60210fe2e1bf0a6697ed7dfd"}
Nov 02 18:11:55.901101 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.901073    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q" event={"ID":"30ef929e-6a26-44e0-a4d6-f376244672a8","Type":"ContainerStarted","Data":"deb4f4febed2d7fd76d6ed2e3825a79f2b406398e08f9d8fad0645aa2d3b176c"}
Nov 02 18:11:55.972567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:55.972517    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q" podStartSLOduration=2.972501911 podStartE2EDuration="2.972501911s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:55.954798292 +0000 UTC m=+2532.445861454" watchObservedRunningTime="2026-11-02 18:11:55.972501911 +0000 UTC m=+2532.463565073"
Nov 02 18:11:56.052661 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.052627    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"]
Nov 02 18:11:56.055588 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.055561    4456 cpu_manager.go:414] "containerMap: removing container" podUID="55d258fd-4f61-4a46-94f6-009c3c370e05" containerName="revoke-keys"
Nov 02 18:11:56.055844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.055802    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="55d258fd-4f61-4a46-94f6-009c3c370e05" containerName="revoke-keys"
Nov 02 18:11:56.056178 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.056160    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="55d258fd-4f61-4a46-94f6-009c3c370e05" containerName="revoke-keys"
Nov 02 18:11:56.074022 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.073976    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"
Nov 02 18:11:56.079178 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.078917    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"]
Nov 02 18:11:56.123843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.115057    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"]
Nov 02 18:11:56.159224 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.159202    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"
Nov 02 18:11:56.160467 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.160447    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"]
Nov 02 18:11:56.175629 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.175576    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95f424e6-fzsg5\" (UID: \"2a8cf6f4-2d7a-4b31-ae04-1192864b3610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"
Nov 02 18:11:56.175726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.175673    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95f424e6-fzsg5\" (UID: \"2a8cf6f4-2d7a-4b31-ae04-1192864b3610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"
Nov 02 18:11:56.175793 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.175768    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lslxw\" (UniqueName: \"kubernetes.io/projected/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-kube-api-access-lslxw\") pod \"maas-api-revoke-sub-95f424e6-fzsg5\" (UID: \"2a8cf6f4-2d7a-4b31-ae04-1192864b3610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"
Nov 02 18:11:56.277416 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.277381    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95f424e6-fzsg5\" (UID: \"2a8cf6f4-2d7a-4b31-ae04-1192864b3610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"
Nov 02 18:11:56.277700 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.277676    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lslxw\" (UniqueName: \"kubernetes.io/projected/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-kube-api-access-lslxw\") pod \"maas-api-revoke-sub-95f424e6-fzsg5\" (UID: \"2a8cf6f4-2d7a-4b31-ae04-1192864b3610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"
Nov 02 18:11:56.277998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.277967    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2rgwd\" (UniqueName: \"kubernetes.io/projected/930e728c-2ee0-4c90-aa3f-59ab9c88a012-kube-api-access-2rgwd\") pod \"maas-api-revoke-sub-6ac0b134-knqf5\" (UID: \"930e728c-2ee0-4c90-aa3f-59ab9c88a012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"
Nov 02 18:11:56.278095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.278035    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930e728c-2ee0-4c90-aa3f-59ab9c88a012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ac0b134-knqf5\" (UID: \"930e728c-2ee0-4c90-aa3f-59ab9c88a012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"
Nov 02 18:11:56.278095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.278068    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930e728c-2ee0-4c90-aa3f-59ab9c88a012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ac0b134-knqf5\" (UID: \"930e728c-2ee0-4c90-aa3f-59ab9c88a012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"
Nov 02 18:11:56.278209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.278105    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95f424e6-fzsg5\" (UID: \"2a8cf6f4-2d7a-4b31-ae04-1192864b3610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"
Nov 02 18:11:56.308842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.279021    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-95f424e6-fzsg5\" (UID: \"2a8cf6f4-2d7a-4b31-ae04-1192864b3610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"
Nov 02 18:11:56.308842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.296322    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" podUID="d837dde2-3939-4400-8ea7-fee58c162983" containerName="payload-processing" containerID="cri-o://17cc61a46dc30e87845b6d1ac12d9771996840ced93cb394cfdbcb29883e8285" gracePeriod=25
Nov 02 18:11:56.308842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.299487    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:56.308842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.299551    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-maas-api-xks-ca\") pod \"maas-api-revoke-sub-95f424e6-fzsg5\" (UID: \"2a8cf6f4-2d7a-4b31-ae04-1192864b3610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"
Nov 02 18:11:56.319802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.318717    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lslxw\" (UniqueName: \"kubernetes.io/projected/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-kube-api-access-lslxw\") pod \"maas-api-revoke-sub-95f424e6-fzsg5\" (UID: \"2a8cf6f4-2d7a-4b31-ae04-1192864b3610\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"
Nov 02 18:11:56.379808 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.379772    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2rgwd\" (UniqueName: \"kubernetes.io/projected/930e728c-2ee0-4c90-aa3f-59ab9c88a012-kube-api-access-2rgwd\") pod \"maas-api-revoke-sub-6ac0b134-knqf5\" (UID: \"930e728c-2ee0-4c90-aa3f-59ab9c88a012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"
Nov 02 18:11:56.403839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.396139    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"
Nov 02 18:11:56.403839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.399914    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930e728c-2ee0-4c90-aa3f-59ab9c88a012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ac0b134-knqf5\" (UID: \"930e728c-2ee0-4c90-aa3f-59ab9c88a012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"
Nov 02 18:11:56.403839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.399964    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930e728c-2ee0-4c90-aa3f-59ab9c88a012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ac0b134-knqf5\" (UID: \"930e728c-2ee0-4c90-aa3f-59ab9c88a012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"
Nov 02 18:11:56.404609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.404539    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930e728c-2ee0-4c90-aa3f-59ab9c88a012-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6ac0b134-knqf5\" (UID: \"930e728c-2ee0-4c90-aa3f-59ab9c88a012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"
Nov 02 18:11:56.417235 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.417198    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930e728c-2ee0-4c90-aa3f-59ab9c88a012-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6ac0b134-knqf5\" (UID: \"930e728c-2ee0-4c90-aa3f-59ab9c88a012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"
Nov 02 18:11:56.421489 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.421411    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2rgwd\" (UniqueName: \"kubernetes.io/projected/930e728c-2ee0-4c90-aa3f-59ab9c88a012-kube-api-access-2rgwd\") pod \"maas-api-revoke-sub-6ac0b134-knqf5\" (UID: \"930e728c-2ee0-4c90-aa3f-59ab9c88a012\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"
Nov 02 18:11:56.434123 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.434097    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"
Nov 02 18:11:56.436065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.436048    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"
Nov 02 18:11:56.440124 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.440108    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"
Nov 02 18:11:56.461187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.460735    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"
Nov 02 18:11:56.466799 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.466765    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:56.471648 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.471616    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"]
Nov 02 18:11:56.479132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.479107    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-b-8a209f-6q8p9"]
Nov 02 18:11:56.506736 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.506658    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-kube-api-access-h8x5n\" (UniqueName: \"kubernetes.io/projected/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-kube-api-access-h8x5n\") pod \"2c58a7cf-d0de-4692-bc36-7d02d346eb2e\" (UID: \"2c58a7cf-d0de-4692-bc36-7d02d346eb2e\") "
Nov 02 18:11:56.508297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.508271    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-maas-api-xks-ca\") pod \"2c58a7cf-d0de-4692-bc36-7d02d346eb2e\" (UID: \"2c58a7cf-d0de-4692-bc36-7d02d346eb2e\") "
Nov 02 18:11:56.509791 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.509697    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-maas-api-openshift-ca" pod "2c58a7cf-d0de-4692-bc36-7d02d346eb2e" (UID: "2c58a7cf-d0de-4692-bc36-7d02d346eb2e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:56.510023 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.509977    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-maas-api-openshift-ca\") pod \"2c58a7cf-d0de-4692-bc36-7d02d346eb2e\" (UID: \"2c58a7cf-d0de-4692-bc36-7d02d346eb2e\") "
Nov 02 18:11:56.510806 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.510769    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:56.528406 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.528356    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-maas-api-xks-ca" pod "2c58a7cf-d0de-4692-bc36-7d02d346eb2e" (UID: "2c58a7cf-d0de-4692-bc36-7d02d346eb2e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:11:56.528509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.528399    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-kube-api-access-h8x5n" pod "2c58a7cf-d0de-4692-bc36-7d02d346eb2e" (UID: "2c58a7cf-d0de-4692-bc36-7d02d346eb2e"). InnerVolumeSpecName "kube-api-access-h8x5n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:56.613252 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.613211    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-maas-api-openshift-ca\") pod \"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc\" (UID: \"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc\") "
Nov 02 18:11:56.613482 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.613342    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d8ec64b7-c3f5-4493-8657-07fe497a12aa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ec64b7-c3f5-4493-8657-07fe497a12aa-maas-api-openshift-ca\") pod \"d8ec64b7-c3f5-4493-8657-07fe497a12aa\" (UID: \"d8ec64b7-c3f5-4493-8657-07fe497a12aa\") "
Nov 02 18:11:56.613563 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.613493    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d8ec64b7-c3f5-4493-8657-07fe497a12aa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ec64b7-c3f5-4493-8657-07fe497a12aa-maas-api-xks-ca\") pod \"d8ec64b7-c3f5-4493-8657-07fe497a12aa\" (UID: \"d8ec64b7-c3f5-4493-8657-07fe497a12aa\") "
Nov 02 18:11:56.613614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.613580    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-kube-api-access-tttcf\" (UniqueName: \"kubernetes.io/projected/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-kube-api-access-tttcf\") pod \"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc\" (UID: \"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc\") "
Nov 02 18:11:56.613724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.613635    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-maas-api-xks-ca\") pod \"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc\" (UID: \"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc\") "
Nov 02 18:11:56.613724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.613692    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d8ec64b7-c3f5-4493-8657-07fe497a12aa-kube-api-access-2qwrd\" (UniqueName: \"kubernetes.io/projected/d8ec64b7-c3f5-4493-8657-07fe497a12aa-kube-api-access-2qwrd\") pod \"d8ec64b7-c3f5-4493-8657-07fe497a12aa\" (UID: \"d8ec64b7-c3f5-4493-8657-07fe497a12aa\") "
Nov 02 18:11:56.614433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.614409    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:56.614527 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.614439    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h8x5n\" (UniqueName: \"kubernetes.io/projected/2c58a7cf-d0de-4692-bc36-7d02d346eb2e-kube-api-access-h8x5n\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:56.615033 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.615007    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-maas-api-openshift-ca" pod "71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc" (UID: "71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:56.615986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.615315    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d8ec64b7-c3f5-4493-8657-07fe497a12aa-maas-api-openshift-ca" pod "d8ec64b7-c3f5-4493-8657-07fe497a12aa" (UID: "d8ec64b7-c3f5-4493-8657-07fe497a12aa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:56.627868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.627690    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:56.633035 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.633010    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d8ec64b7-c3f5-4493-8657-07fe497a12aa-maas-api-xks-ca" pod "d8ec64b7-c3f5-4493-8657-07fe497a12aa" (UID: "d8ec64b7-c3f5-4493-8657-07fe497a12aa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:11:56.633444 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.633415    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-kube-api-access-tttcf" pod "71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc" (UID: "71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc"). InnerVolumeSpecName "kube-api-access-tttcf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:56.633954 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.633749    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d8ec64b7-c3f5-4493-8657-07fe497a12aa-kube-api-access-2qwrd" pod "d8ec64b7-c3f5-4493-8657-07fe497a12aa" (UID: "d8ec64b7-c3f5-4493-8657-07fe497a12aa"). InnerVolumeSpecName "kube-api-access-2qwrd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:56.644922 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.644867    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-maas-api-xks-ca" pod "71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc" (UID: "71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:11:56.715726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.715632    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:56.715726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.715665    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d8ec64b7-c3f5-4493-8657-07fe497a12aa-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:56.715726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.715680    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d8ec64b7-c3f5-4493-8657-07fe497a12aa-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:56.715726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.715689    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tttcf\" (UniqueName: \"kubernetes.io/projected/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-kube-api-access-tttcf\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:56.715726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.715698    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:56.715726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.715705    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2qwrd\" (UniqueName: \"kubernetes.io/projected/d8ec64b7-c3f5-4493-8657-07fe497a12aa-kube-api-access-2qwrd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:56.747949 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.747922    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 18:11:56.811789 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.805882    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:56.833732 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:56.833700    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod930e728c_2ee0_4c90_aa3f_59ab9c88a012.slice/crio-6c8d497f99a1b5c6223d822e8993ec2e79cfb441e110b7339e071df0cd8126fc WatchSource:0}: Error finding container 6c8d497f99a1b5c6223d822e8993ec2e79cfb441e110b7339e071df0cd8126fc: Status 404 returned error can't find the container with id 6c8d497f99a1b5c6223d822e8993ec2e79cfb441e110b7339e071df0cd8126fc
Nov 02 18:11:56.835537 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.835511    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"]
Nov 02 18:11:56.910984 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.910943    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"]
Nov 02 18:11:56.915407 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:56.915373    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod2a8cf6f4_2d7a_4b31_ae04_1192864b3610.slice/crio-71228147ce77cd7768d6016a124ccf60454c7cafafe988d4ab92cd3565d318f8 WatchSource:0}: Error finding container 71228147ce77cd7768d6016a124ccf60454c7cafafe988d4ab92cd3565d318f8: Status 404 returned error can't find the container with id 71228147ce77cd7768d6016a124ccf60454c7cafafe988d4ab92cd3565d318f8
Nov 02 18:11:56.917729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.917713    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"
Nov 02 18:11:56.918039 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.917758    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb" event={"ID":"2c58a7cf-d0de-4692-bc36-7d02d346eb2e","Type":"ContainerDied","Data":"2a4420ece27d37555913f1620cb5c1626a524a45cf2daf2659ca5f748800d9ab"}
Nov 02 18:11:56.918039 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.918011    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2a4420ece27d37555913f1620cb5c1626a524a45cf2daf2659ca5f748800d9ab"
Nov 02 18:11:56.920126 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.920108    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d837dde2-3939-4400-8ea7-fee58c162983-kube-api-access-vj66v\" (UniqueName: \"kubernetes.io/projected/d837dde2-3939-4400-8ea7-fee58c162983-kube-api-access-vj66v\") pod \"d837dde2-3939-4400-8ea7-fee58c162983\" (UID: \"d837dde2-3939-4400-8ea7-fee58c162983\") "
Nov 02 18:11:56.920212 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.920197    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d837dde2-3939-4400-8ea7-fee58c162983-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d837dde2-3939-4400-8ea7-fee58c162983-plugins-config-volume\") pod \"d837dde2-3939-4400-8ea7-fee58c162983\" (UID: \"d837dde2-3939-4400-8ea7-fee58c162983\") "
Nov 02 18:11:56.920971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.920949    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d837dde2-3939-4400-8ea7-fee58c162983-plugins-config-volume" pod "d837dde2-3939-4400-8ea7-fee58c162983" (UID: "d837dde2-3939-4400-8ea7-fee58c162983"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:56.921119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.921055    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr" event={"ID":"d8ec64b7-c3f5-4493-8657-07fe497a12aa","Type":"ContainerDied","Data":"c660b5af7306d9c04dd3b0e08f6d6bca442d8402b6363871efddf11a602035a6"}
Nov 02 18:11:56.921119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.921069    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"
Nov 02 18:11:56.921119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.921098    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c660b5af7306d9c04dd3b0e08f6d6bca442d8402b6363871efddf11a602035a6"
Nov 02 18:11:56.922962 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.922920    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d837dde2-3939-4400-8ea7-fee58c162983-kube-api-access-vj66v" pod "d837dde2-3939-4400-8ea7-fee58c162983" (UID: "d837dde2-3939-4400-8ea7-fee58c162983"). InnerVolumeSpecName "kube-api-access-vj66v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:56.923296 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.923277    4456 generic.go:356] "Generic (PLEG): container finished" podID="30ef929e-6a26-44e0-a4d6-f376244672a8" containerID="0d54b34fabde9aeebc7f496fae68a4cd7e94840f60210fe2e1bf0a6697ed7dfd" exitCode=0
Nov 02 18:11:56.923384 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.923359    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q" event={"ID":"30ef929e-6a26-44e0-a4d6-f376244672a8","Type":"ContainerDied","Data":"0d54b34fabde9aeebc7f496fae68a4cd7e94840f60210fe2e1bf0a6697ed7dfd"}
Nov 02 18:11:56.925696 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.925669    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j" event={"ID":"71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc","Type":"ContainerDied","Data":"5fa680781ccf2aabe1b1b25efb418dd360f8dddd0f3bec5482c22e420d654be0"}
Nov 02 18:11:56.925923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.925893    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5fa680781ccf2aabe1b1b25efb418dd360f8dddd0f3bec5482c22e420d654be0"
Nov 02 18:11:56.926006 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.925704    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"
Nov 02 18:11:56.943775 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.943752    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5" event={"ID":"930e728c-2ee0-4c90-aa3f-59ab9c88a012","Type":"ContainerStarted","Data":"6c8d497f99a1b5c6223d822e8993ec2e79cfb441e110b7339e071df0cd8126fc"}
Nov 02 18:11:56.945512 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.945483    4456 generic.go:356] "Generic (PLEG): container finished" podID="d837dde2-3939-4400-8ea7-fee58c162983" containerID="17cc61a46dc30e87845b6d1ac12d9771996840ced93cb394cfdbcb29883e8285" exitCode=0
Nov 02 18:11:56.945801 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.945784    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"
Nov 02 18:11:56.946970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.946945    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" event={"ID":"d837dde2-3939-4400-8ea7-fee58c162983","Type":"ContainerDied","Data":"17cc61a46dc30e87845b6d1ac12d9771996840ced93cb394cfdbcb29883e8285"}
Nov 02 18:11:56.947067 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.946993    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx" event={"ID":"d837dde2-3939-4400-8ea7-fee58c162983","Type":"ContainerDied","Data":"06c8e45e6152b3243b57e5eff85877ff6128978eed88c9a57087d53841ecd736"}
Nov 02 18:11:56.947067 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.947020    4456 scope.go:122] "RemoveContainer" containerID="17cc61a46dc30e87845b6d1ac12d9771996840ced93cb394cfdbcb29883e8285"
Nov 02 18:11:56.965361 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.965324    4456 scope.go:122] "RemoveContainer" containerID="17cc61a46dc30e87845b6d1ac12d9771996840ced93cb394cfdbcb29883e8285"
Nov 02 18:11:56.966644 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:11:56.966623    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"17cc61a46dc30e87845b6d1ac12d9771996840ced93cb394cfdbcb29883e8285\": container with ID starting with 17cc61a46dc30e87845b6d1ac12d9771996840ced93cb394cfdbcb29883e8285 not found: ID does not exist" containerID="17cc61a46dc30e87845b6d1ac12d9771996840ced93cb394cfdbcb29883e8285"
Nov 02 18:11:56.966792 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.966744    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"17cc61a46dc30e87845b6d1ac12d9771996840ced93cb394cfdbcb29883e8285"} err="failed to get container status \"17cc61a46dc30e87845b6d1ac12d9771996840ced93cb394cfdbcb29883e8285\": rpc error: code = NotFound desc = could not find container \"17cc61a46dc30e87845b6d1ac12d9771996840ced93cb394cfdbcb29883e8285\": container with ID starting with 17cc61a46dc30e87845b6d1ac12d9771996840ced93cb394cfdbcb29883e8285 not found: ID does not exist"
Nov 02 18:11:56.978770 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:56.978740    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:57.022091 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.022061    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vj66v\" (UniqueName: \"kubernetes.io/projected/d837dde2-3939-4400-8ea7-fee58c162983-kube-api-access-vj66v\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:57.022211 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.022093    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/d837dde2-3939-4400-8ea7-fee58c162983-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:57.059537 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.059504    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"]
Nov 02 18:11:57.062738 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.062679    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w6-b91583-7c4bf96889-qvslx"]
Nov 02 18:11:57.136074 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.136041    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"
Nov 02 18:11:57.226856 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.224205    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a230f129-4b8a-4544-9e35-053804d1edd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a230f129-4b8a-4544-9e35-053804d1edd7-maas-api-xks-ca\") pod \"a230f129-4b8a-4544-9e35-053804d1edd7\" (UID: \"a230f129-4b8a-4544-9e35-053804d1edd7\") "
Nov 02 18:11:57.226856 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.224344    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a230f129-4b8a-4544-9e35-053804d1edd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a230f129-4b8a-4544-9e35-053804d1edd7-maas-api-openshift-ca\") pod \"a230f129-4b8a-4544-9e35-053804d1edd7\" (UID: \"a230f129-4b8a-4544-9e35-053804d1edd7\") "
Nov 02 18:11:57.226856 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.224439    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a230f129-4b8a-4544-9e35-053804d1edd7-kube-api-access-p7fjh\" (UniqueName: \"kubernetes.io/projected/a230f129-4b8a-4544-9e35-053804d1edd7-kube-api-access-p7fjh\") pod \"a230f129-4b8a-4544-9e35-053804d1edd7\" (UID: \"a230f129-4b8a-4544-9e35-053804d1edd7\") "
Nov 02 18:11:57.226856 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.226218    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a230f129-4b8a-4544-9e35-053804d1edd7-maas-api-openshift-ca" pod "a230f129-4b8a-4544-9e35-053804d1edd7" (UID: "a230f129-4b8a-4544-9e35-053804d1edd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:57.227658 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.227630    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a230f129-4b8a-4544-9e35-053804d1edd7-maas-api-xks-ca" pod "a230f129-4b8a-4544-9e35-053804d1edd7" (UID: "a230f129-4b8a-4544-9e35-053804d1edd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:11:57.228180 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.228156    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a230f129-4b8a-4544-9e35-053804d1edd7-kube-api-access-p7fjh" pod "a230f129-4b8a-4544-9e35-053804d1edd7" (UID: "a230f129-4b8a-4544-9e35-053804d1edd7"). InnerVolumeSpecName "kube-api-access-p7fjh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:57.325902 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.325791    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a230f129-4b8a-4544-9e35-053804d1edd7-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:57.325902 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.325878    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a230f129-4b8a-4544-9e35-053804d1edd7-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:57.325902 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.325889    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p7fjh\" (UniqueName: \"kubernetes.io/projected/a230f129-4b8a-4544-9e35-053804d1edd7-kube-api-access-p7fjh\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:57.622577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.622486    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8908e429-6925-4fc6-a3f0-6fbae9fa0d24" path="/var/lib/kubelet/pods/8908e429-6925-4fc6-a3f0-6fbae9fa0d24/volumes"
Nov 02 18:11:57.623009 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.622995    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d837dde2-3939-4400-8ea7-fee58c162983" path="/var/lib/kubelet/pods/d837dde2-3939-4400-8ea7-fee58c162983/volumes"
Nov 02 18:11:57.772006 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.771972    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:57.919518 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.919480    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:57.955909 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.955865    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5" event={"ID":"930e728c-2ee0-4c90-aa3f-59ab9c88a012","Type":"ContainerStarted","Data":"1005e3ad004e76a6d733579931a774a5011da75161fb0c4938ed55c9328c5248"}
Nov 02 18:11:57.957340 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.957321    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"
Nov 02 18:11:57.957445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.957351    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr" event={"ID":"a230f129-4b8a-4544-9e35-053804d1edd7","Type":"ContainerDied","Data":"881b8d217e29c0cafa90204803f47557a2733f7b68c4b58a1328caad43471411"}
Nov 02 18:11:57.957445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.957398    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="881b8d217e29c0cafa90204803f47557a2733f7b68c4b58a1328caad43471411"
Nov 02 18:11:57.960341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.960315    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5" event={"ID":"2a8cf6f4-2d7a-4b31-ae04-1192864b3610","Type":"ContainerStarted","Data":"9fd4431a4f8d70b237e6855e1443cdcf43cce8d0e7028815878dab823ab38898"}
Nov 02 18:11:57.960434 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:57.960358    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5" event={"ID":"2a8cf6f4-2d7a-4b31-ae04-1192864b3610","Type":"ContainerStarted","Data":"71228147ce77cd7768d6016a124ccf60454c7cafafe988d4ab92cd3565d318f8"}
Nov 02 18:11:58.022369 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.022320    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5" podStartSLOduration=2.022304789 podStartE2EDuration="2.022304789s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:58.020613532 +0000 UTC m=+2534.511676718" watchObservedRunningTime="2026-11-02 18:11:58.022304789 +0000 UTC m=+2534.513367929"
Nov 02 18:11:58.115087 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.115025    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5" podStartSLOduration=2.1150098489999998 podStartE2EDuration="2.115009849s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:11:58.112323989 +0000 UTC m=+2534.603387171" watchObservedRunningTime="2026-11-02 18:11:58.115009849 +0000 UTC m=+2534.606073010"
Nov 02 18:11:58.177924 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.160374    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"]
Nov 02 18:11:58.177924 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.160619    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy" containerID="cri-o://20b2d3a75919af0722d71b55fa5ed703b6ecda4bdc3481dee9e990f775021807" gracePeriod=30
Nov 02 18:11:58.254654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.254621    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:58.321919 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.319688    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"
Nov 02 18:11:58.439320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.439292    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/30ef929e-6a26-44e0-a4d6-f376244672a8-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ef929e-6a26-44e0-a4d6-f376244672a8-maas-api-xks-ca\") pod \"30ef929e-6a26-44e0-a4d6-f376244672a8\" (UID: \"30ef929e-6a26-44e0-a4d6-f376244672a8\") "
Nov 02 18:11:58.439495 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.439366    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/30ef929e-6a26-44e0-a4d6-f376244672a8-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ef929e-6a26-44e0-a4d6-f376244672a8-maas-api-openshift-ca\") pod \"30ef929e-6a26-44e0-a4d6-f376244672a8\" (UID: \"30ef929e-6a26-44e0-a4d6-f376244672a8\") "
Nov 02 18:11:58.439495 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.439415    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/30ef929e-6a26-44e0-a4d6-f376244672a8-kube-api-access-8pcdd\" (UniqueName: \"kubernetes.io/projected/30ef929e-6a26-44e0-a4d6-f376244672a8-kube-api-access-8pcdd\") pod \"30ef929e-6a26-44e0-a4d6-f376244672a8\" (UID: \"30ef929e-6a26-44e0-a4d6-f376244672a8\") "
Nov 02 18:11:58.439789 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.439766    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/30ef929e-6a26-44e0-a4d6-f376244672a8-maas-api-openshift-ca" pod "30ef929e-6a26-44e0-a4d6-f376244672a8" (UID: "30ef929e-6a26-44e0-a4d6-f376244672a8"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:11:58.439975 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.439958    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/30ef929e-6a26-44e0-a4d6-f376244672a8-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:58.441692 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.441668    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/30ef929e-6a26-44e0-a4d6-f376244672a8-kube-api-access-8pcdd" pod "30ef929e-6a26-44e0-a4d6-f376244672a8" (UID: "30ef929e-6a26-44e0-a4d6-f376244672a8"). InnerVolumeSpecName "kube-api-access-8pcdd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:11:58.441783 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.441691    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/30ef929e-6a26-44e0-a4d6-f376244672a8-maas-api-xks-ca" pod "30ef929e-6a26-44e0-a4d6-f376244672a8" (UID: "30ef929e-6a26-44e0-a4d6-f376244672a8"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:11:58.541421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.541384    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/30ef929e-6a26-44e0-a4d6-f376244672a8-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:58.541558 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.541427    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8pcdd\" (UniqueName: \"kubernetes.io/projected/30ef929e-6a26-44e0-a4d6-f376244672a8-kube-api-access-8pcdd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:11:58.650336 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.650289    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:58.801685 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.799341    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:11:58.850283 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.850253    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"]
Nov 02 18:11:58.850869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.850829    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d837dde2-3939-4400-8ea7-fee58c162983" containerName="payload-processing"
Nov 02 18:11:58.850869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.850845    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d837dde2-3939-4400-8ea7-fee58c162983" containerName="payload-processing"
Nov 02 18:11:58.850869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.850866    4456 cpu_manager.go:414] "containerMap: removing container" podUID="71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc" containerName="revoke-keys"
Nov 02 18:11:58.850869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.850871    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc" containerName="revoke-keys"
Nov 02 18:11:58.851029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.850882    4456 cpu_manager.go:414] "containerMap: removing container" podUID="30ef929e-6a26-44e0-a4d6-f376244672a8" containerName="revoke-keys"
Nov 02 18:11:58.851029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.850888    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="30ef929e-6a26-44e0-a4d6-f376244672a8" containerName="revoke-keys"
Nov 02 18:11:58.851029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.850900    4456 cpu_manager.go:414] "containerMap: removing container" podUID="2c58a7cf-d0de-4692-bc36-7d02d346eb2e" containerName="revoke-keys"
Nov 02 18:11:58.851029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.850905    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2c58a7cf-d0de-4692-bc36-7d02d346eb2e" containerName="revoke-keys"
Nov 02 18:11:58.851029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.850923    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d8ec64b7-c3f5-4493-8657-07fe497a12aa" containerName="revoke-keys"
Nov 02 18:11:58.851029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.850927    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d8ec64b7-c3f5-4493-8657-07fe497a12aa" containerName="revoke-keys"
Nov 02 18:11:58.851029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.850948    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a230f129-4b8a-4544-9e35-053804d1edd7" containerName="revoke-keys"
Nov 02 18:11:58.851029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.850953    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a230f129-4b8a-4544-9e35-053804d1edd7" containerName="revoke-keys"
Nov 02 18:11:58.851029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.851028    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a230f129-4b8a-4544-9e35-053804d1edd7" containerName="revoke-keys"
Nov 02 18:11:58.851248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.851036    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d837dde2-3939-4400-8ea7-fee58c162983" containerName="payload-processing"
Nov 02 18:11:58.851248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.851047    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc" containerName="revoke-keys"
Nov 02 18:11:58.851248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.851052    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="30ef929e-6a26-44e0-a4d6-f376244672a8" containerName="revoke-keys"
Nov 02 18:11:58.851248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.851059    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="2c58a7cf-d0de-4692-bc36-7d02d346eb2e" containerName="revoke-keys"
Nov 02 18:11:58.851248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.851066    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d8ec64b7-c3f5-4493-8657-07fe497a12aa" containerName="revoke-keys"
Nov 02 18:11:58.854472 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.854456    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"
Nov 02 18:11:58.874042 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.874017    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"]
Nov 02 18:11:58.949033 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.948999    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq6l6\" (UniqueName: \"kubernetes.io/projected/e2442a9b-67dd-4a54-bd70-c9075e1a6522-kube-api-access-jq6l6\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv\" (UID: \"e2442a9b-67dd-4a54-bd70-c9075e1a6522\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"
Nov 02 18:11:58.949213 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.949057    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2442a9b-67dd-4a54-bd70-c9075e1a6522-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv\" (UID: \"e2442a9b-67dd-4a54-bd70-c9075e1a6522\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"
Nov 02 18:11:58.949213 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.949168    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2442a9b-67dd-4a54-bd70-c9075e1a6522-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv\" (UID: \"e2442a9b-67dd-4a54-bd70-c9075e1a6522\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"
Nov 02 18:11:58.970683 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.970654    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q" event={"ID":"30ef929e-6a26-44e0-a4d6-f376244672a8","Type":"ContainerDied","Data":"deb4f4febed2d7fd76d6ed2e3825a79f2b406398e08f9d8fad0645aa2d3b176c"}
Nov 02 18:11:58.970868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.970709    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="deb4f4febed2d7fd76d6ed2e3825a79f2b406398e08f9d8fad0645aa2d3b176c"
Nov 02 18:11:58.970868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.970662    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"
Nov 02 18:11:58.972458 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.972430    4456 generic.go:356] "Generic (PLEG): container finished" podID="2a8cf6f4-2d7a-4b31-ae04-1192864b3610" containerID="9fd4431a4f8d70b237e6855e1443cdcf43cce8d0e7028815878dab823ab38898" exitCode=0
Nov 02 18:11:58.972575 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.972510    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5" event={"ID":"2a8cf6f4-2d7a-4b31-ae04-1192864b3610","Type":"ContainerDied","Data":"9fd4431a4f8d70b237e6855e1443cdcf43cce8d0e7028815878dab823ab38898"}
Nov 02 18:11:58.974545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.974522    4456 generic.go:356] "Generic (PLEG): container finished" podID="930e728c-2ee0-4c90-aa3f-59ab9c88a012" containerID="1005e3ad004e76a6d733579931a774a5011da75161fb0c4938ed55c9328c5248" exitCode=0
Nov 02 18:11:58.974646 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:58.974602    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5" event={"ID":"930e728c-2ee0-4c90-aa3f-59ab9c88a012","Type":"ContainerDied","Data":"1005e3ad004e76a6d733579931a774a5011da75161fb0c4938ed55c9328c5248"}
Nov 02 18:11:59.050721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:59.050691    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq6l6\" (UniqueName: \"kubernetes.io/projected/e2442a9b-67dd-4a54-bd70-c9075e1a6522-kube-api-access-jq6l6\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv\" (UID: \"e2442a9b-67dd-4a54-bd70-c9075e1a6522\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"
Nov 02 18:11:59.051143 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:59.050765    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2442a9b-67dd-4a54-bd70-c9075e1a6522-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv\" (UID: \"e2442a9b-67dd-4a54-bd70-c9075e1a6522\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"
Nov 02 18:11:59.051143 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:59.050839    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2442a9b-67dd-4a54-bd70-c9075e1a6522-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv\" (UID: \"e2442a9b-67dd-4a54-bd70-c9075e1a6522\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"
Nov 02 18:11:59.051533 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:59.051509    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2442a9b-67dd-4a54-bd70-c9075e1a6522-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv\" (UID: \"e2442a9b-67dd-4a54-bd70-c9075e1a6522\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"
Nov 02 18:11:59.053631 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:59.053564    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2442a9b-67dd-4a54-bd70-c9075e1a6522-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv\" (UID: \"e2442a9b-67dd-4a54-bd70-c9075e1a6522\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"
Nov 02 18:11:59.082723 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:59.082696    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq6l6\" (UniqueName: \"kubernetes.io/projected/e2442a9b-67dd-4a54-bd70-c9075e1a6522-kube-api-access-jq6l6\") pod \"maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv\" (UID: \"e2442a9b-67dd-4a54-bd70-c9075e1a6522\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"
Nov 02 18:11:59.169614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:59.169581    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"
Nov 02 18:11:59.476921 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:11:59.476889    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode2442a9b_67dd_4a54_bd70_c9075e1a6522.slice/crio-4a8d9a24c04802b196561b1bea466766bccbc4480f963ee7ef8a991b63a1c171 WatchSource:0}: Error finding container 4a8d9a24c04802b196561b1bea466766bccbc4480f963ee7ef8a991b63a1c171: Status 404 returned error can't find the container with id 4a8d9a24c04802b196561b1bea466766bccbc4480f963ee7ef8a991b63a1c171
Nov 02 18:11:59.477395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:59.477371    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"]
Nov 02 18:11:59.986577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:59.986532    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv" event={"ID":"e2442a9b-67dd-4a54-bd70-c9075e1a6522","Type":"ContainerStarted","Data":"fca79505dcc07d2a45ecf9c33d89ff089634354e7540cedb2dca53cbb393a362"}
Nov 02 18:11:59.986769 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:11:59.986608    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv" event={"ID":"e2442a9b-67dd-4a54-bd70-c9075e1a6522","Type":"ContainerStarted","Data":"4a8d9a24c04802b196561b1bea466766bccbc4480f963ee7ef8a991b63a1c171"}
Nov 02 18:12:00.039664 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.039596    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv" podStartSLOduration=2.039573917 podStartE2EDuration="2.039573917s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:11:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:12:00.033746724 +0000 UTC m=+2536.524809886" watchObservedRunningTime="2026-11-02 18:12:00.039573917 +0000 UTC m=+2536.530637079"
Nov 02 18:12:00.202385 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.202355    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"
Nov 02 18:12:00.242704 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.242676    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"
Nov 02 18:12:00.367210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.367129    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-maas-api-openshift-ca\") pod \"2a8cf6f4-2d7a-4b31-ae04-1192864b3610\" (UID: \"2a8cf6f4-2d7a-4b31-ae04-1192864b3610\") "
Nov 02 18:12:00.367346 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.367299    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/930e728c-2ee0-4c90-aa3f-59ab9c88a012-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930e728c-2ee0-4c90-aa3f-59ab9c88a012-maas-api-xks-ca\") pod \"930e728c-2ee0-4c90-aa3f-59ab9c88a012\" (UID: \"930e728c-2ee0-4c90-aa3f-59ab9c88a012\") "
Nov 02 18:12:00.367346 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.367338    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-kube-api-access-lslxw\" (UniqueName: \"kubernetes.io/projected/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-kube-api-access-lslxw\") pod \"2a8cf6f4-2d7a-4b31-ae04-1192864b3610\" (UID: \"2a8cf6f4-2d7a-4b31-ae04-1192864b3610\") "
Nov 02 18:12:00.367446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.367373    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-maas-api-xks-ca\") pod \"2a8cf6f4-2d7a-4b31-ae04-1192864b3610\" (UID: \"2a8cf6f4-2d7a-4b31-ae04-1192864b3610\") "
Nov 02 18:12:00.367516 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.367497    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/930e728c-2ee0-4c90-aa3f-59ab9c88a012-kube-api-access-2rgwd\" (UniqueName: \"kubernetes.io/projected/930e728c-2ee0-4c90-aa3f-59ab9c88a012-kube-api-access-2rgwd\") pod \"930e728c-2ee0-4c90-aa3f-59ab9c88a012\" (UID: \"930e728c-2ee0-4c90-aa3f-59ab9c88a012\") "
Nov 02 18:12:00.367557 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.367512    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-maas-api-openshift-ca" pod "2a8cf6f4-2d7a-4b31-ae04-1192864b3610" (UID: "2a8cf6f4-2d7a-4b31-ae04-1192864b3610"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:00.367597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.367585    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/930e728c-2ee0-4c90-aa3f-59ab9c88a012-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930e728c-2ee0-4c90-aa3f-59ab9c88a012-maas-api-openshift-ca\") pod \"930e728c-2ee0-4c90-aa3f-59ab9c88a012\" (UID: \"930e728c-2ee0-4c90-aa3f-59ab9c88a012\") "
Nov 02 18:12:00.369065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.368993    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:00.369341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.369231    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/930e728c-2ee0-4c90-aa3f-59ab9c88a012-maas-api-openshift-ca" pod "930e728c-2ee0-4c90-aa3f-59ab9c88a012" (UID: "930e728c-2ee0-4c90-aa3f-59ab9c88a012"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:00.371131 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.371109    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-kube-api-access-lslxw" pod "2a8cf6f4-2d7a-4b31-ae04-1192864b3610" (UID: "2a8cf6f4-2d7a-4b31-ae04-1192864b3610"). InnerVolumeSpecName "kube-api-access-lslxw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:00.371237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.371149    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-maas-api-xks-ca" pod "2a8cf6f4-2d7a-4b31-ae04-1192864b3610" (UID: "2a8cf6f4-2d7a-4b31-ae04-1192864b3610"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:00.371441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.371421    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/930e728c-2ee0-4c90-aa3f-59ab9c88a012-kube-api-access-2rgwd" pod "930e728c-2ee0-4c90-aa3f-59ab9c88a012" (UID: "930e728c-2ee0-4c90-aa3f-59ab9c88a012"). InnerVolumeSpecName "kube-api-access-2rgwd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:00.371630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.371612    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/930e728c-2ee0-4c90-aa3f-59ab9c88a012-maas-api-xks-ca" pod "930e728c-2ee0-4c90-aa3f-59ab9c88a012" (UID: "930e728c-2ee0-4c90-aa3f-59ab9c88a012"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:00.471326 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.471284    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/930e728c-2ee0-4c90-aa3f-59ab9c88a012-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:00.471326 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.471330    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lslxw\" (UniqueName: \"kubernetes.io/projected/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-kube-api-access-lslxw\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:00.471494 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.471347    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2a8cf6f4-2d7a-4b31-ae04-1192864b3610-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:00.471494 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.471362    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rgwd\" (UniqueName: \"kubernetes.io/projected/930e728c-2ee0-4c90-aa3f-59ab9c88a012-kube-api-access-2rgwd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:00.471494 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.471377    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/930e728c-2ee0-4c90-aa3f-59ab9c88a012-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:00.545249 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.545207    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main" containerID="cri-o://57939473b6ad2b6cbdafcfe2737dd5cf2ffeefee3affd12a79845d1dc1dea2f7" gracePeriod=45
Nov 02 18:12:00.547987 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.547947    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main" containerID="cri-o://764623736da34bdff5ec0d93b5511172b9e7e7729309eb85acb60f34966d682b" gracePeriod=45
Nov 02 18:12:00.552596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.552436    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" containerID="cri-o://1cbb553419640bc05c0058aa78a8963cbe07f8e1708345f3d87e0220045f11b2" gracePeriod=45
Nov 02 18:12:00.655041 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.654961    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:00.894006 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.893971    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 18:12:00.926428 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.926395    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 18:12:00.945523 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:00.945035    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 18:12:01.048151 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.048106    4456 generic.go:356] "Generic (PLEG): container finished" podID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerID="57939473b6ad2b6cbdafcfe2737dd5cf2ffeefee3affd12a79845d1dc1dea2f7" exitCode=0
Nov 02 18:12:01.048323 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.048229    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" event={"ID":"57a419b6-5333-4b2c-9f01-051dc82c8707","Type":"ContainerDied","Data":"57939473b6ad2b6cbdafcfe2737dd5cf2ffeefee3affd12a79845d1dc1dea2f7"}
Nov 02 18:12:01.048323 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.048293    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9" event={"ID":"57a419b6-5333-4b2c-9f01-051dc82c8707","Type":"ContainerDied","Data":"99dcea58b362ef916eb88c98ae6818f61e3f222684ff2c8d11b46cd3b7133bbf"}
Nov 02 18:12:01.048323 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.048315    4456 scope.go:122] "RemoveContainer" containerID="57939473b6ad2b6cbdafcfe2737dd5cf2ffeefee3affd12a79845d1dc1dea2f7"
Nov 02 18:12:01.048506 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.048494    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"
Nov 02 18:12:01.051639 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.051460    4456 generic.go:356] "Generic (PLEG): container finished" podID="e2442a9b-67dd-4a54-bd70-c9075e1a6522" containerID="fca79505dcc07d2a45ecf9c33d89ff089634354e7540cedb2dca53cbb393a362" exitCode=0
Nov 02 18:12:01.051639 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.051628    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv" event={"ID":"e2442a9b-67dd-4a54-bd70-c9075e1a6522","Type":"ContainerDied","Data":"fca79505dcc07d2a45ecf9c33d89ff089634354e7540cedb2dca53cbb393a362"}
Nov 02 18:12:01.055860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.055396    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5" event={"ID":"2a8cf6f4-2d7a-4b31-ae04-1192864b3610","Type":"ContainerDied","Data":"71228147ce77cd7768d6016a124ccf60454c7cafafe988d4ab92cd3565d318f8"}
Nov 02 18:12:01.055860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.055453    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="71228147ce77cd7768d6016a124ccf60454c7cafafe988d4ab92cd3565d318f8"
Nov 02 18:12:01.055860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.055544    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"
Nov 02 18:12:01.062074 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.062016    4456 generic.go:356] "Generic (PLEG): container finished" podID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerID="1cbb553419640bc05c0058aa78a8963cbe07f8e1708345f3d87e0220045f11b2" exitCode=0
Nov 02 18:12:01.062184 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.062161    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" event={"ID":"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34","Type":"ContainerDied","Data":"1cbb553419640bc05c0058aa78a8963cbe07f8e1708345f3d87e0220045f11b2"}
Nov 02 18:12:01.062262 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.062195    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" event={"ID":"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34","Type":"ContainerDied","Data":"08103e3b7cf6a358f4d2a88299f447d12720c89aa2456f9bec55e46dc101805e"}
Nov 02 18:12:01.062336 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.062280    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"
Nov 02 18:12:01.064780 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.064752    4456 generic.go:356] "Generic (PLEG): container finished" podID="5734524d-67b2-4e24-a836-3ac078d5a271" containerID="764623736da34bdff5ec0d93b5511172b9e7e7729309eb85acb60f34966d682b" exitCode=0
Nov 02 18:12:01.064893 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.064849    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" event={"ID":"5734524d-67b2-4e24-a836-3ac078d5a271","Type":"ContainerDied","Data":"764623736da34bdff5ec0d93b5511172b9e7e7729309eb85acb60f34966d682b"}
Nov 02 18:12:01.064893 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.064878    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b" event={"ID":"5734524d-67b2-4e24-a836-3ac078d5a271","Type":"ContainerDied","Data":"73d2c013b474c6ff1af7f2714382fc5912e92bcd9b6bb918cb97fe9b6363856b"}
Nov 02 18:12:01.065023 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.064954    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"
Nov 02 18:12:01.069561 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.067128    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5" event={"ID":"930e728c-2ee0-4c90-aa3f-59ab9c88a012","Type":"ContainerDied","Data":"6c8d497f99a1b5c6223d822e8993ec2e79cfb441e110b7339e071df0cd8126fc"}
Nov 02 18:12:01.069561 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.067178    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6c8d497f99a1b5c6223d822e8993ec2e79cfb441e110b7339e071df0cd8126fc"
Nov 02 18:12:01.069561 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.067250    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"
Nov 02 18:12:01.083875 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.083854    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5734524d-67b2-4e24-a836-3ac078d5a271-tls-certs\" (UniqueName: \"kubernetes.io/secret/5734524d-67b2-4e24-a836-3ac078d5a271-tls-certs\") pod \"5734524d-67b2-4e24-a836-3ac078d5a271\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") "
Nov 02 18:12:01.084064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084038    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-tmp-dir\") pod \"5734524d-67b2-4e24-a836-3ac078d5a271\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") "
Nov 02 18:12:01.084172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084139    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-tls-certs\" (UniqueName: \"kubernetes.io/secret/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-tls-certs\") pod \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") "
Nov 02 18:12:01.084298 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084238    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-kube-api-access-lr7dd\" (UniqueName: \"kubernetes.io/projected/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-kube-api-access-lr7dd\") pod \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") "
Nov 02 18:12:01.084298 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084289    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/57a419b6-5333-4b2c-9f01-051dc82c8707-tls-certs\" (UniqueName: \"kubernetes.io/secret/57a419b6-5333-4b2c-9f01-051dc82c8707-tls-certs\") pod \"57a419b6-5333-4b2c-9f01-051dc82c8707\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") "
Nov 02 18:12:01.084386 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084323    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-tmp-dir\") pod \"57a419b6-5333-4b2c-9f01-051dc82c8707\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") "
Nov 02 18:12:01.084438 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084404    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-dshm\") pod \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") "
Nov 02 18:12:01.084438 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084410    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-tmp-dir" pod "5734524d-67b2-4e24-a836-3ac078d5a271" (UID: "5734524d-67b2-4e24-a836-3ac078d5a271"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.084529 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084433    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-model-cache\") pod \"57a419b6-5333-4b2c-9f01-051dc82c8707\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") "
Nov 02 18:12:01.084529 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084486    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/57a419b6-5333-4b2c-9f01-051dc82c8707-kube-api-access-lr9ws\" (UniqueName: \"kubernetes.io/projected/57a419b6-5333-4b2c-9f01-051dc82c8707-kube-api-access-lr9ws\") pod \"57a419b6-5333-4b2c-9f01-051dc82c8707\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") "
Nov 02 18:12:01.084610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084537    4456 scope.go:122] "RemoveContainer" containerID="57939473b6ad2b6cbdafcfe2737dd5cf2ffeefee3affd12a79845d1dc1dea2f7"
Nov 02 18:12:01.084610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084591    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-tmp-dir\") pod \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") "
Nov 02 18:12:01.084698 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084632    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-model-cache\") pod \"5734524d-67b2-4e24-a836-3ac078d5a271\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") "
Nov 02 18:12:01.084698 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084665    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/5734524d-67b2-4e24-a836-3ac078d5a271-kube-api-access-4dtmz\" (UniqueName: \"kubernetes.io/projected/5734524d-67b2-4e24-a836-3ac078d5a271-kube-api-access-4dtmz\") pod \"5734524d-67b2-4e24-a836-3ac078d5a271\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") "
Nov 02 18:12:01.084788 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084701    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-home\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-home\") pod \"5734524d-67b2-4e24-a836-3ac078d5a271\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") "
Nov 02 18:12:01.084788 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084750    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-dshm\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-dshm\") pod \"57a419b6-5333-4b2c-9f01-051dc82c8707\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") "
Nov 02 18:12:01.084909 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084802    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-home\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-home\") pod \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") "
Nov 02 18:12:01.084909 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084858    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-model-cache\") pod \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\" (UID: \"5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34\") "
Nov 02 18:12:01.084909 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084875    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-tmp-dir" pod "57a419b6-5333-4b2c-9f01-051dc82c8707" (UID: "57a419b6-5333-4b2c-9f01-051dc82c8707"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.085001 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.084981    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-home\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-home\") pod \"57a419b6-5333-4b2c-9f01-051dc82c8707\" (UID: \"57a419b6-5333-4b2c-9f01-051dc82c8707\") "
Nov 02 18:12:01.085045 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.085024    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-dshm\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-dshm\") pod \"5734524d-67b2-4e24-a836-3ac078d5a271\" (UID: \"5734524d-67b2-4e24-a836-3ac078d5a271\") "
Nov 02 18:12:01.085376 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.085344    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-tmp-dir" pod "5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" (UID: "5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.085963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.085932    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-home" pod "5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" (UID: "5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.086196 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.086164    4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-tmp-dir\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.086240 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.086194    4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-tmp-dir\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.086240 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.086209    4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-home\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.086240 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.086224    4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-tmp-dir\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.086854 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.086809    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-home" pod "5734524d-67b2-4e24-a836-3ac078d5a271" (UID: "5734524d-67b2-4e24-a836-3ac078d5a271"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.087993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.087965    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-model-cache" pod "5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" (UID: "5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.088799 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.088774    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-home" pod "57a419b6-5333-4b2c-9f01-051dc82c8707" (UID: "57a419b6-5333-4b2c-9f01-051dc82c8707"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.091201 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:01.091017    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"57939473b6ad2b6cbdafcfe2737dd5cf2ffeefee3affd12a79845d1dc1dea2f7\": container with ID starting with 57939473b6ad2b6cbdafcfe2737dd5cf2ffeefee3affd12a79845d1dc1dea2f7 not found: ID does not exist" containerID="57939473b6ad2b6cbdafcfe2737dd5cf2ffeefee3affd12a79845d1dc1dea2f7"
Nov 02 18:12:01.091201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.091064    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"57939473b6ad2b6cbdafcfe2737dd5cf2ffeefee3affd12a79845d1dc1dea2f7"} err="failed to get container status \"57939473b6ad2b6cbdafcfe2737dd5cf2ffeefee3affd12a79845d1dc1dea2f7\": rpc error: code = NotFound desc = could not find container \"57939473b6ad2b6cbdafcfe2737dd5cf2ffeefee3affd12a79845d1dc1dea2f7\": container with ID starting with 57939473b6ad2b6cbdafcfe2737dd5cf2ffeefee3affd12a79845d1dc1dea2f7 not found: ID does not exist"
Nov 02 18:12:01.091201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.091093    4456 scope.go:122] "RemoveContainer" containerID="1cbb553419640bc05c0058aa78a8963cbe07f8e1708345f3d87e0220045f11b2"
Nov 02 18:12:01.091406 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.091287    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-model-cache" pod "57a419b6-5333-4b2c-9f01-051dc82c8707" (UID: "57a419b6-5333-4b2c-9f01-051dc82c8707"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.091406 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.091288    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-model-cache" pod "5734524d-67b2-4e24-a836-3ac078d5a271" (UID: "5734524d-67b2-4e24-a836-3ac078d5a271"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.114951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.114911    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/57a419b6-5333-4b2c-9f01-051dc82c8707-tls-certs" pod "57a419b6-5333-4b2c-9f01-051dc82c8707" (UID: "57a419b6-5333-4b2c-9f01-051dc82c8707"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:01.126322 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.126280    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5734524d-67b2-4e24-a836-3ac078d5a271-tls-certs" pod "5734524d-67b2-4e24-a836-3ac078d5a271" (UID: "5734524d-67b2-4e24-a836-3ac078d5a271"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:01.126322 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.126311    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-kube-api-access-lr7dd" pod "5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" (UID: "5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34"). InnerVolumeSpecName "kube-api-access-lr7dd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:01.126536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.126340    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/5734524d-67b2-4e24-a836-3ac078d5a271-kube-api-access-4dtmz" pod "5734524d-67b2-4e24-a836-3ac078d5a271" (UID: "5734524d-67b2-4e24-a836-3ac078d5a271"). InnerVolumeSpecName "kube-api-access-4dtmz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:01.126536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.126375    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-tls-certs" pod "5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" (UID: "5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:01.126536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.126388    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-dshm" pod "5734524d-67b2-4e24-a836-3ac078d5a271" (UID: "5734524d-67b2-4e24-a836-3ac078d5a271"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.126536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.126425    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-dshm" pod "5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" (UID: "5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.126536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.126436    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-dshm" pod "57a419b6-5333-4b2c-9f01-051dc82c8707" (UID: "57a419b6-5333-4b2c-9f01-051dc82c8707"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:01.135619 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.135589    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/57a419b6-5333-4b2c-9f01-051dc82c8707-kube-api-access-lr9ws" pod "57a419b6-5333-4b2c-9f01-051dc82c8707" (UID: "57a419b6-5333-4b2c-9f01-051dc82c8707"). InnerVolumeSpecName "kube-api-access-lr9ws". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:01.157801 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.157774    4456 scope.go:122] "RemoveContainer" containerID="1cbb553419640bc05c0058aa78a8963cbe07f8e1708345f3d87e0220045f11b2"
Nov 02 18:12:01.158233 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:01.158211    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1cbb553419640bc05c0058aa78a8963cbe07f8e1708345f3d87e0220045f11b2\": container with ID starting with 1cbb553419640bc05c0058aa78a8963cbe07f8e1708345f3d87e0220045f11b2 not found: ID does not exist" containerID="1cbb553419640bc05c0058aa78a8963cbe07f8e1708345f3d87e0220045f11b2"
Nov 02 18:12:01.158332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.158241    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1cbb553419640bc05c0058aa78a8963cbe07f8e1708345f3d87e0220045f11b2"} err="failed to get container status \"1cbb553419640bc05c0058aa78a8963cbe07f8e1708345f3d87e0220045f11b2\": rpc error: code = NotFound desc = could not find container \"1cbb553419640bc05c0058aa78a8963cbe07f8e1708345f3d87e0220045f11b2\": container with ID starting with 1cbb553419640bc05c0058aa78a8963cbe07f8e1708345f3d87e0220045f11b2 not found: ID does not exist"
Nov 02 18:12:01.158332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.158258    4456 scope.go:122] "RemoveContainer" containerID="764623736da34bdff5ec0d93b5511172b9e7e7729309eb85acb60f34966d682b"
Nov 02 18:12:01.187284 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.187257    4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-model-cache\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.187284 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.187280    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4dtmz\" (UniqueName: \"kubernetes.io/projected/5734524d-67b2-4e24-a836-3ac078d5a271-kube-api-access-4dtmz\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.187442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.187291    4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-home\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.187442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.187301    4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-dshm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.187442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.187309    4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-model-cache\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.187442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.187317    4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-home\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.187442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.187325    4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5734524d-67b2-4e24-a836-3ac078d5a271-dshm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.187442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.187333    4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5734524d-67b2-4e24-a836-3ac078d5a271-tls-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.187442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.187341    4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-tls-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.187442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.187349    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr7dd\" (UniqueName: \"kubernetes.io/projected/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-kube-api-access-lr7dd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.187790 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.187357    4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/57a419b6-5333-4b2c-9f01-051dc82c8707-tls-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.187790 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.187365    4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34-dshm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.187790 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.187375    4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/57a419b6-5333-4b2c-9f01-051dc82c8707-model-cache\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.187790 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.187382    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lr9ws\" (UniqueName: \"kubernetes.io/projected/57a419b6-5333-4b2c-9f01-051dc82c8707-kube-api-access-lr9ws\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:01.201087 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.201061    4456 scope.go:122] "RemoveContainer" containerID="764623736da34bdff5ec0d93b5511172b9e7e7729309eb85acb60f34966d682b"
Nov 02 18:12:01.201434 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:01.201410    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"764623736da34bdff5ec0d93b5511172b9e7e7729309eb85acb60f34966d682b\": container with ID starting with 764623736da34bdff5ec0d93b5511172b9e7e7729309eb85acb60f34966d682b not found: ID does not exist" containerID="764623736da34bdff5ec0d93b5511172b9e7e7729309eb85acb60f34966d682b"
Nov 02 18:12:01.201501 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.201447    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"764623736da34bdff5ec0d93b5511172b9e7e7729309eb85acb60f34966d682b"} err="failed to get container status \"764623736da34bdff5ec0d93b5511172b9e7e7729309eb85acb60f34966d682b\": rpc error: code = NotFound desc = could not find container \"764623736da34bdff5ec0d93b5511172b9e7e7729309eb85acb60f34966d682b\": container with ID starting with 764623736da34bdff5ec0d93b5511172b9e7e7729309eb85acb60f34966d682b not found: ID does not exist"
Nov 02 18:12:01.404568 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.404535    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"]
Nov 02 18:12:01.407971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.407944    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-120394/distinct-2-w1-120394-kserve-766594cbd8-r7nr9"]
Nov 02 18:12:01.445905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.445825    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"]
Nov 02 18:12:01.451052 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.451026    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-120394/simulator-w1-120394-kserve-6979bf4f4f-scg5b"]
Nov 02 18:12:01.488190 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.488156    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"]
Nov 02 18:12:01.493386 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.493363    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s"]
Nov 02 18:12:01.623176 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.623138    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" path="/var/lib/kubelet/pods/5734524d-67b2-4e24-a836-3ac078d5a271/volumes"
Nov 02 18:12:01.623829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.623795    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" path="/var/lib/kubelet/pods/57a419b6-5333-4b2c-9f01-051dc82c8707/volumes"
Nov 02 18:12:01.633571 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.633539    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" path="/var/lib/kubelet/pods/5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34/volumes"
Nov 02 18:12:01.887777 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.887703    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"]
Nov 02 18:12:01.888258 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.888245    4456 cpu_manager.go:414] "containerMap: removing container" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main"
Nov 02 18:12:01.888297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.888259    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main"
Nov 02 18:12:01.888297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.888290    4456 cpu_manager.go:414] "containerMap: removing container" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main"
Nov 02 18:12:01.888297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.888295    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main"
Nov 02 18:12:01.888391 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.888302    4456 cpu_manager.go:414] "containerMap: removing container" podUID="930e728c-2ee0-4c90-aa3f-59ab9c88a012" containerName="revoke-keys"
Nov 02 18:12:01.888391 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.888308    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="930e728c-2ee0-4c90-aa3f-59ab9c88a012" containerName="revoke-keys"
Nov 02 18:12:01.888391 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.888316    4456 cpu_manager.go:414] "containerMap: removing container" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main"
Nov 02 18:12:01.888391 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.888321    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main"
Nov 02 18:12:01.888391 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.888344    4456 cpu_manager.go:414] "containerMap: removing container" podUID="2a8cf6f4-2d7a-4b31-ae04-1192864b3610" containerName="revoke-keys"
Nov 02 18:12:01.888391 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.888349    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2a8cf6f4-2d7a-4b31-ae04-1192864b3610" containerName="revoke-keys"
Nov 02 18:12:01.888538 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.888423    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="2a8cf6f4-2d7a-4b31-ae04-1192864b3610" containerName="revoke-keys"
Nov 02 18:12:01.888538 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.888434    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="57a419b6-5333-4b2c-9f01-051dc82c8707" containerName="main"
Nov 02 18:12:01.888538 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.888445    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main"
Nov 02 18:12:01.888538 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.888453    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="930e728c-2ee0-4c90-aa3f-59ab9c88a012" containerName="revoke-keys"
Nov 02 18:12:01.888538 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.888459    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="5734524d-67b2-4e24-a836-3ac078d5a271" containerName="main"
Nov 02 18:12:01.904293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.904256    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w1-120394/premium-w1-120394-kserve-669cf87f88-rmj4s" podUID="5835a6e8-f2ee-4dd4-9afe-ad0f7a013e34" containerName="main" probeResult="failure" output="Get \"https://10.134.0.50:8000/ready\": context deadline exceeded"
Nov 02 18:12:01.917483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.917455    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"]
Nov 02 18:12:01.917623 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.917561    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:01.933799 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.933770    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-tls\""
Nov 02 18:12:01.934046 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.934022    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istio-kubeconfig\""
Nov 02 18:12:01.934147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.934129    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"istiod-openshift-gateway-dockercfg-dklz6\""
Nov 02 18:12:01.934218 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:01.934159    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"cacerts\""
Nov 02 18:12:02.001577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.001540    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.001756 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.001588    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.001756 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.001723    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ppv2m\" (UniqueName: \"kubernetes.io/projected/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-kube-api-access-ppv2m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.001898 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.001760    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.001898 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.001785    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.001993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.001920    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.001993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.001954    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.102572 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.102545    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.102731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.102598    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.102731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.102643    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.102731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.102675    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ppv2m\" (UniqueName: \"kubernetes.io/projected/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-kube-api-access-ppv2m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.102731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.102707    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.103006 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.102739    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.103006 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.102881    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.103463 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.103432    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-csr-ca-configmap\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.106206 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.106004    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-cacerts\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.106206 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.106162    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-csr-dns-cert\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.108269 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.108241    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-local-certs\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.108655 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.108630    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-kubeconfig\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.131536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.131504    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-token\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.132123 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.132099    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ppv2m\" (UniqueName: \"kubernetes.io/projected/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-kube-api-access-ppv2m\") pod \"istiod-openshift-gateway-7ddd9cfdc8-6bvgj\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") " pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.253802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.253768    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:02.271013 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.270990    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"
Nov 02 18:12:02.304863 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.304832    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e2442a9b-67dd-4a54-bd70-c9075e1a6522-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2442a9b-67dd-4a54-bd70-c9075e1a6522-maas-api-openshift-ca\") pod \"e2442a9b-67dd-4a54-bd70-c9075e1a6522\" (UID: \"e2442a9b-67dd-4a54-bd70-c9075e1a6522\") "
Nov 02 18:12:02.304986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.304930    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e2442a9b-67dd-4a54-bd70-c9075e1a6522-kube-api-access-jq6l6\" (UniqueName: \"kubernetes.io/projected/e2442a9b-67dd-4a54-bd70-c9075e1a6522-kube-api-access-jq6l6\") pod \"e2442a9b-67dd-4a54-bd70-c9075e1a6522\" (UID: \"e2442a9b-67dd-4a54-bd70-c9075e1a6522\") "
Nov 02 18:12:02.304986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.304962    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e2442a9b-67dd-4a54-bd70-c9075e1a6522-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2442a9b-67dd-4a54-bd70-c9075e1a6522-maas-api-xks-ca\") pod \"e2442a9b-67dd-4a54-bd70-c9075e1a6522\" (UID: \"e2442a9b-67dd-4a54-bd70-c9075e1a6522\") "
Nov 02 18:12:02.305188 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.305161    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e2442a9b-67dd-4a54-bd70-c9075e1a6522-maas-api-openshift-ca" pod "e2442a9b-67dd-4a54-bd70-c9075e1a6522" (UID: "e2442a9b-67dd-4a54-bd70-c9075e1a6522"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:02.305472 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.305449    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e2442a9b-67dd-4a54-bd70-c9075e1a6522-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:02.307754 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.307725    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e2442a9b-67dd-4a54-bd70-c9075e1a6522-kube-api-access-jq6l6" pod "e2442a9b-67dd-4a54-bd70-c9075e1a6522" (UID: "e2442a9b-67dd-4a54-bd70-c9075e1a6522"). InnerVolumeSpecName "kube-api-access-jq6l6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:02.309104 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.309083    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e2442a9b-67dd-4a54-bd70-c9075e1a6522-maas-api-xks-ca" pod "e2442a9b-67dd-4a54-bd70-c9075e1a6522" (UID: "e2442a9b-67dd-4a54-bd70-c9075e1a6522"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:02.406930 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.406841    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq6l6\" (UniqueName: \"kubernetes.io/projected/e2442a9b-67dd-4a54-bd70-c9075e1a6522-kube-api-access-jq6l6\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:02.406930 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.406893    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e2442a9b-67dd-4a54-bd70-c9075e1a6522-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:02.616503 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:12:02.616464    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod3ffa306f_b84b_4bb9_ba77_e0bdaf58b777.slice/crio-b32c466b30fabb8ae5aabb7783a3233427c97e8dff62c70d6f8f394e4408d4c0 WatchSource:0}: Error finding container b32c466b30fabb8ae5aabb7783a3233427c97e8dff62c70d6f8f394e4408d4c0: Status 404 returned error can't find the container with id b32c466b30fabb8ae5aabb7783a3233427c97e8dff62c70d6f8f394e4408d4c0
Nov 02 18:12:02.617057 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.617030    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"]
Nov 02 18:12:02.619329 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.619295    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:12:02.619433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.619398    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:12:02.747798 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:02.747668    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:03.092622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.090601    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"
Nov 02 18:12:03.092622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.091900    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv" event={"ID":"e2442a9b-67dd-4a54-bd70-c9075e1a6522","Type":"ContainerDied","Data":"4a8d9a24c04802b196561b1bea466766bccbc4480f963ee7ef8a991b63a1c171"}
Nov 02 18:12:03.092622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.091960    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a8d9a24c04802b196561b1bea466766bccbc4480f963ee7ef8a991b63a1c171"
Nov 02 18:12:03.093793 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.093768    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj" event={"ID":"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777","Type":"ContainerStarted","Data":"09bb8945087fb15758d5d9efebc5a08418929a47a699e984e7be398d49c03d69"}
Nov 02 18:12:03.094204 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.094101    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj" event={"ID":"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777","Type":"ContainerStarted","Data":"b32c466b30fabb8ae5aabb7783a3233427c97e8dff62c70d6f8f394e4408d4c0"}
Nov 02 18:12:03.094204 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.094160    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:03.095950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.095923    4456 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6bvgj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.250:8080/ready\": dial tcp 10.134.0.250:8080: connect: connection refused" start-of-body=
Nov 02 18:12:03.096061 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.095989    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj" podUID="3ffa306f-b84b-4bb9-ba77-e0bdaf58b777" containerName="discovery" probeResult="failure" output="Get \"http://10.134.0.250:8080/ready\": dial tcp 10.134.0.250:8080: connect: connection refused"
Nov 02 18:12:03.147495 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.147446    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj" podStartSLOduration=2.147433072 podStartE2EDuration="2.147433072s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:12:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:12:03.144347285 +0000 UTC m=+2539.635410554" watchObservedRunningTime="2026-11-02 18:12:03.147433072 +0000 UTC m=+2539.638496233"
Nov 02 18:12:03.551208 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.551182    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:12:03.626485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.625964    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/92ad0220-435f-4ae8-b743-3421ce98e6fb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92ad0220-435f-4ae8-b743-3421ce98e6fb-istiod-ca-cert\") pod \"92ad0220-435f-4ae8-b743-3421ce98e6fb\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") "
Nov 02 18:12:03.626485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.626035    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ad0220-435f-4ae8-b743-3421ce98e6fb-kube-api-access-2rrvm\" (UniqueName: \"kubernetes.io/projected/92ad0220-435f-4ae8-b743-3421ce98e6fb-kube-api-access-2rrvm\") pod \"92ad0220-435f-4ae8-b743-3421ce98e6fb\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") "
Nov 02 18:12:03.626485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.626086    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-podinfo\") pod \"92ad0220-435f-4ae8-b743-3421ce98e6fb\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") "
Nov 02 18:12:03.626485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.626185    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-workload-socket\") pod \"92ad0220-435f-4ae8-b743-3421ce98e6fb\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") "
Nov 02 18:12:03.626485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.626259    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-data\") pod \"92ad0220-435f-4ae8-b743-3421ce98e6fb\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") "
Nov 02 18:12:03.626887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.626312    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-token\" (UniqueName: \"kubernetes.io/projected/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-token\") pod \"92ad0220-435f-4ae8-b743-3421ce98e6fb\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") "
Nov 02 18:12:03.626887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.626344    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-credential-socket\") pod \"92ad0220-435f-4ae8-b743-3421ce98e6fb\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") "
Nov 02 18:12:03.626887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.626452    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-envoy\") pod \"92ad0220-435f-4ae8-b743-3421ce98e6fb\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") "
Nov 02 18:12:03.626887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.626528    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-workload-certs\") pod \"92ad0220-435f-4ae8-b743-3421ce98e6fb\" (UID: \"92ad0220-435f-4ae8-b743-3421ce98e6fb\") "
Nov 02 18:12:03.627331 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.627126    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-data" pod "92ad0220-435f-4ae8-b743-3421ce98e6fb" (UID: "92ad0220-435f-4ae8-b743-3421ce98e6fb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:03.628441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.627556    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/92ad0220-435f-4ae8-b743-3421ce98e6fb-istiod-ca-cert" pod "92ad0220-435f-4ae8-b743-3421ce98e6fb" (UID: "92ad0220-435f-4ae8-b743-3421ce98e6fb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:03.628441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.627806    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-credential-socket" pod "92ad0220-435f-4ae8-b743-3421ce98e6fb" (UID: "92ad0220-435f-4ae8-b743-3421ce98e6fb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:03.634026 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.631118    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-workload-socket" pod "92ad0220-435f-4ae8-b743-3421ce98e6fb" (UID: "92ad0220-435f-4ae8-b743-3421ce98e6fb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:03.636310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.635888    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-workload-certs" pod "92ad0220-435f-4ae8-b743-3421ce98e6fb" (UID: "92ad0220-435f-4ae8-b743-3421ce98e6fb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:03.637637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.637592    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-envoy" pod "92ad0220-435f-4ae8-b743-3421ce98e6fb" (UID: "92ad0220-435f-4ae8-b743-3421ce98e6fb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:03.639741 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.639713    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-podinfo" pod "92ad0220-435f-4ae8-b743-3421ce98e6fb" (UID: "92ad0220-435f-4ae8-b743-3421ce98e6fb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:12:03.640263 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.640235    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-token" pod "92ad0220-435f-4ae8-b743-3421ce98e6fb" (UID: "92ad0220-435f-4ae8-b743-3421ce98e6fb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:03.640690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.640646    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/92ad0220-435f-4ae8-b743-3421ce98e6fb-kube-api-access-2rrvm" pod "92ad0220-435f-4ae8-b743-3421ce98e6fb" (UID: "92ad0220-435f-4ae8-b743-3421ce98e6fb"). InnerVolumeSpecName "kube-api-access-2rrvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:03.728306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.728264    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:03.728306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.728308    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:03.728306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.728324    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:03.728632 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.728340    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:03.728632 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.728358    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:03.728632 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.728373    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/92ad0220-435f-4ae8-b743-3421ce98e6fb-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:03.728632 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.728386    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/92ad0220-435f-4ae8-b743-3421ce98e6fb-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:03.728632 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.728401    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2rrvm\" (UniqueName: \"kubernetes.io/projected/92ad0220-435f-4ae8-b743-3421ce98e6fb-kube-api-access-2rrvm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:03.728632 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:03.728416    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/92ad0220-435f-4ae8-b743-3421ce98e6fb-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:04.109448 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.109156    4456 generic.go:356] "Generic (PLEG): container finished" podID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerID="20b2d3a75919af0722d71b55fa5ed703b6ecda4bdc3481dee9e990f775021807" exitCode=0
Nov 02 18:12:04.109448 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.109219    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"
Nov 02 18:12:04.109448 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.109266    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" event={"ID":"92ad0220-435f-4ae8-b743-3421ce98e6fb","Type":"ContainerDied","Data":"20b2d3a75919af0722d71b55fa5ed703b6ecda4bdc3481dee9e990f775021807"}
Nov 02 18:12:04.109448 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.109325    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4" event={"ID":"92ad0220-435f-4ae8-b743-3421ce98e6fb","Type":"ContainerDied","Data":"2b799dfc861353931928f0368501ec5932f9339bcb26dea69afe9cd496f226f9"}
Nov 02 18:12:04.109448 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.109355    4456 scope.go:122] "RemoveContainer" containerID="20b2d3a75919af0722d71b55fa5ed703b6ecda4bdc3481dee9e990f775021807"
Nov 02 18:12:04.114768 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.114732    4456 generic.go:356] "Generic (PLEG): container finished" podID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerID="51e90f82024911642a387ee990edbbcce91cbeeb7879f1ae0c57ac3433a8cfc7" exitCode=0
Nov 02 18:12:04.116430 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.116400    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" event={"ID":"856c2dca-bad2-4f62-a536-1ca0e09f0a12","Type":"ContainerDied","Data":"51e90f82024911642a387ee990edbbcce91cbeeb7879f1ae0c57ac3433a8cfc7"}
Nov 02 18:12:04.117070 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.117044    4456 scope.go:122] "RemoveContainer" containerID="51e90f82024911642a387ee990edbbcce91cbeeb7879f1ae0c57ac3433a8cfc7"
Nov 02 18:12:04.117896 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.117860    4456 patch_prober.go:29] interesting pod/istiod-openshift-gateway-7ddd9cfdc8-6bvgj container/discovery namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:12:04.118117 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.118090    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj" podUID="3ffa306f-b84b-4bb9-ba77-e0bdaf58b777" containerName="discovery" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:12:04.120240 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.120203    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:12:04.120333 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.120310    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:12:04.120410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.120364    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:12:04.129317 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.129296    4456 scope.go:122] "RemoveContainer" containerID="20b2d3a75919af0722d71b55fa5ed703b6ecda4bdc3481dee9e990f775021807"
Nov 02 18:12:04.129851 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:04.129722    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20b2d3a75919af0722d71b55fa5ed703b6ecda4bdc3481dee9e990f775021807\": container with ID starting with 20b2d3a75919af0722d71b55fa5ed703b6ecda4bdc3481dee9e990f775021807 not found: ID does not exist" containerID="20b2d3a75919af0722d71b55fa5ed703b6ecda4bdc3481dee9e990f775021807"
Nov 02 18:12:04.129851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.129749    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20b2d3a75919af0722d71b55fa5ed703b6ecda4bdc3481dee9e990f775021807"} err="failed to get container status \"20b2d3a75919af0722d71b55fa5ed703b6ecda4bdc3481dee9e990f775021807\": rpc error: code = NotFound desc = could not find container \"20b2d3a75919af0722d71b55fa5ed703b6ecda4bdc3481dee9e990f775021807\": container with ID starting with 20b2d3a75919af0722d71b55fa5ed703b6ecda4bdc3481dee9e990f775021807 not found: ID does not exist"
Nov 02 18:12:04.218643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.218554    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:04.228913 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.228865    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"]
Nov 02 18:12:04.240509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.240478    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-inf-b-b2fc9f-openshift-default-76856594bf-tcxc4"]
Nov 02 18:12:04.377259 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.377178    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:04.493840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.490149    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"]
Nov 02 18:12:04.551051 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.551020    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"]
Nov 02 18:12:04.555630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.555595    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:04.726188 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.726158    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:04.873996 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:04.873960    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:05.028839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.028736    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:05.127179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.127147    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" event={"ID":"856c2dca-bad2-4f62-a536-1ca0e09f0a12","Type":"ContainerStarted","Data":"43b444ecc72b8f3eef31fafa69b45fd05eaa0fc026c197971824e9e9553d8fe9"}
Nov 02 18:12:05.194922 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.179892    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:05.272467 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.272430    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"]
Nov 02 18:12:05.276657 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.276625    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-872354-ppd4j"]
Nov 02 18:12:05.508839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.508136    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:05.621915 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.621876    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc" path="/var/lib/kubelet/pods/71f8e0ab-cd0d-42ad-a56b-f8c22c7208cc/volumes"
Nov 02 18:12:05.622425 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.622406    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" path="/var/lib/kubelet/pods/92ad0220-435f-4ae8-b743-3421ce98e6fb/volumes"
Nov 02 18:12:05.659017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.658987    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:05.730359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.730317    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"]
Nov 02 18:12:05.743076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.743047    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a5c1da-94w58"]
Nov 02 18:12:05.818897 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.818782    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:05.851070 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.851031    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:12:05.851225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.851193    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 18:12:05.851271 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.851246    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 18:12:05.965493 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:05.965460    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:06.461955 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.461923    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"]
Nov 02 18:12:06.462458 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.462444    4456 cpu_manager.go:414] "containerMap: removing container" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy"
Nov 02 18:12:06.462515 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.462459    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy"
Nov 02 18:12:06.462515 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.462503    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e2442a9b-67dd-4a54-bd70-c9075e1a6522" containerName="revoke-keys"
Nov 02 18:12:06.462515 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.462509    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e2442a9b-67dd-4a54-bd70-c9075e1a6522" containerName="revoke-keys"
Nov 02 18:12:06.462601 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.462584    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e2442a9b-67dd-4a54-bd70-c9075e1a6522" containerName="revoke-keys"
Nov 02 18:12:06.462601 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.462595    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="92ad0220-435f-4ae8-b743-3421ce98e6fb" containerName="istio-proxy"
Nov 02 18:12:06.465833 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.465799    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"
Nov 02 18:12:06.511946 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.511909    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"]
Nov 02 18:12:06.512211 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.512193    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:12:06.512984 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.512961    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:12:06.605520 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.605485    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e869a01e-b2dc-414c-949b-f86d71c428be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp\" (UID: \"e869a01e-b2dc-414c-949b-f86d71c428be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"
Nov 02 18:12:06.605701 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.605526    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e869a01e-b2dc-414c-949b-f86d71c428be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp\" (UID: \"e869a01e-b2dc-414c-949b-f86d71c428be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"
Nov 02 18:12:06.605746 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.605689    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-p6vwh\" (UniqueName: \"kubernetes.io/projected/e869a01e-b2dc-414c-949b-f86d71c428be-kube-api-access-p6vwh\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp\" (UID: \"e869a01e-b2dc-414c-949b-f86d71c428be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"
Nov 02 18:12:06.707506 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.707469    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e869a01e-b2dc-414c-949b-f86d71c428be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp\" (UID: \"e869a01e-b2dc-414c-949b-f86d71c428be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"
Nov 02 18:12:06.707506 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.707512    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e869a01e-b2dc-414c-949b-f86d71c428be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp\" (UID: \"e869a01e-b2dc-414c-949b-f86d71c428be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"
Nov 02 18:12:06.707969 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.707577    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-p6vwh\" (UniqueName: \"kubernetes.io/projected/e869a01e-b2dc-414c-949b-f86d71c428be-kube-api-access-p6vwh\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp\" (UID: \"e869a01e-b2dc-414c-949b-f86d71c428be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"
Nov 02 18:12:06.708344 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.708324    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e869a01e-b2dc-414c-949b-f86d71c428be-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp\" (UID: \"e869a01e-b2dc-414c-949b-f86d71c428be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"
Nov 02 18:12:06.710131 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.710107    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e869a01e-b2dc-414c-949b-f86d71c428be-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp\" (UID: \"e869a01e-b2dc-414c-949b-f86d71c428be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"
Nov 02 18:12:06.732611 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.732543    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-p6vwh\" (UniqueName: \"kubernetes.io/projected/e869a01e-b2dc-414c-949b-f86d71c428be-kube-api-access-p6vwh\") pod \"maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp\" (UID: \"e869a01e-b2dc-414c-949b-f86d71c428be\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"
Nov 02 18:12:06.793840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.784014    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"
Nov 02 18:12:06.851670 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.851615    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 18:12:06.852011 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:06.851958    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 18:12:07.009916 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:07.007406    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:07.118148 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:07.118120    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:12:07.148388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:07.148360    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"]
Nov 02 18:12:07.152303 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:12:07.152269    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pode869a01e_b2dc_414c_949b_f86d71c428be.slice/crio-53b5afd84af0711011c0219889632712acbf19f3b7e46215a8f31b72b1783c72 WatchSource:0}: Error finding container 53b5afd84af0711011c0219889632712acbf19f3b7e46215a8f31b72b1783c72: Status 404 returned error can't find the container with id 53b5afd84af0711011c0219889632712acbf19f3b7e46215a8f31b72b1783c72
Nov 02 18:12:07.618643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:07.618608    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="55d258fd-4f61-4a46-94f6-009c3c370e05" path="/var/lib/kubelet/pods/55d258fd-4f61-4a46-94f6-009c3c370e05/volumes"
Nov 02 18:12:07.851995 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:07.851957    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 18:12:07.852416 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:07.852008    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 18:12:07.891545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:07.891481    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:08.156949 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:08.156859    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp" event={"ID":"e869a01e-b2dc-414c-949b-f86d71c428be","Type":"ContainerStarted","Data":"852b8f642a03ab7a55b5e40b2fec8845a457e699441ea49ae7a5e4e8e30b2e12"}
Nov 02 18:12:08.156949 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:08.156914    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp" event={"ID":"e869a01e-b2dc-414c-949b-f86d71c428be","Type":"ContainerStarted","Data":"53b5afd84af0711011c0219889632712acbf19f3b7e46215a8f31b72b1783c72"}
Nov 02 18:12:08.210842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:08.209259    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp" podStartSLOduration=2.209240187 podStartE2EDuration="2.209240187s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:12:06 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:12:08.20836238 +0000 UTC m=+2544.699425541" watchObservedRunningTime="2026-11-02 18:12:08.209240187 +0000 UTC m=+2544.700303350"
Nov 02 18:12:08.851768 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:08.851734    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 18:12:08.851965 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:08.851792    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 18:12:09.168323 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:09.168230    4456 generic.go:356] "Generic (PLEG): container finished" podID="e869a01e-b2dc-414c-949b-f86d71c428be" containerID="852b8f642a03ab7a55b5e40b2fec8845a457e699441ea49ae7a5e4e8e30b2e12" exitCode=0
Nov 02 18:12:09.168706 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:09.168329    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp" event={"ID":"e869a01e-b2dc-414c-949b-f86d71c428be","Type":"ContainerDied","Data":"852b8f642a03ab7a55b5e40b2fec8845a457e699441ea49ae7a5e4e8e30b2e12"}
Nov 02 18:12:09.490933 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:09.490886    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing" containerID="cri-o://65dc1c7055ca0c593790169b343803715577f608817bf5ba500657f16af44d0b" gracePeriod=25
Nov 02 18:12:09.552235 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:09.552190    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing" containerID="cri-o://fb0803029471e70042acd1e45626801c9ff68565bd8c57f88a3acdbb82e2b010" gracePeriod=25
Nov 02 18:12:09.712837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:09.712786    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"]
Nov 02 18:12:09.813681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:09.813587    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:09.851806 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:09.851776    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused" start-of-body=
Nov 02 18:12:09.851806 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:09.851846    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.229:15021/healthz/ready\": dial tcp 10.134.0.229:15021: connect: connection refused"
Nov 02 18:12:09.976365 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:09.976333    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:10.131443 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.131358    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:10.240936 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.240897    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"]
Nov 02 18:12:10.245143 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.245120    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"
Nov 02 18:12:10.266217 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.266187    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"]
Nov 02 18:12:10.364116 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.364083    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8045e931-2116-422f-806e-f18e3b757a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv\" (UID: \"8045e931-2116-422f-806e-f18e3b757a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"
Nov 02 18:12:10.364287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.364171    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8045e931-2116-422f-806e-f18e3b757a0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv\" (UID: \"8045e931-2116-422f-806e-f18e3b757a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"
Nov 02 18:12:10.364287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.364273    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g87vd\" (UniqueName: \"kubernetes.io/projected/8045e931-2116-422f-806e-f18e3b757a0c-kube-api-access-g87vd\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv\" (UID: \"8045e931-2116-422f-806e-f18e3b757a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"
Nov 02 18:12:10.465385 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.465350    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8045e931-2116-422f-806e-f18e3b757a0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv\" (UID: \"8045e931-2116-422f-806e-f18e3b757a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"
Nov 02 18:12:10.465535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.465506    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g87vd\" (UniqueName: \"kubernetes.io/projected/8045e931-2116-422f-806e-f18e3b757a0c-kube-api-access-g87vd\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv\" (UID: \"8045e931-2116-422f-806e-f18e3b757a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"
Nov 02 18:12:10.466020 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.465991    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8045e931-2116-422f-806e-f18e3b757a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv\" (UID: \"8045e931-2116-422f-806e-f18e3b757a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"
Nov 02 18:12:10.466261 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.466176    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"
Nov 02 18:12:10.466731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.466706    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8045e931-2116-422f-806e-f18e3b757a0c-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv\" (UID: \"8045e931-2116-422f-806e-f18e3b757a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"
Nov 02 18:12:10.468440 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.468416    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8045e931-2116-422f-806e-f18e3b757a0c-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv\" (UID: \"8045e931-2116-422f-806e-f18e3b757a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"
Nov 02 18:12:10.489983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.489954    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g87vd\" (UniqueName: \"kubernetes.io/projected/8045e931-2116-422f-806e-f18e3b757a0c-kube-api-access-g87vd\") pod \"maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv\" (UID: \"8045e931-2116-422f-806e-f18e3b757a0c\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"
Nov 02 18:12:10.559240 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.559212    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"
Nov 02 18:12:10.566856 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.566826    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e869a01e-b2dc-414c-949b-f86d71c428be-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e869a01e-b2dc-414c-949b-f86d71c428be-maas-api-openshift-ca\") pod \"e869a01e-b2dc-414c-949b-f86d71c428be\" (UID: \"e869a01e-b2dc-414c-949b-f86d71c428be\") "
Nov 02 18:12:10.566992 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.566963    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e869a01e-b2dc-414c-949b-f86d71c428be-kube-api-access-p6vwh\" (UniqueName: \"kubernetes.io/projected/e869a01e-b2dc-414c-949b-f86d71c428be-kube-api-access-p6vwh\") pod \"e869a01e-b2dc-414c-949b-f86d71c428be\" (UID: \"e869a01e-b2dc-414c-949b-f86d71c428be\") "
Nov 02 18:12:10.567066 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.567029    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/e869a01e-b2dc-414c-949b-f86d71c428be-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e869a01e-b2dc-414c-949b-f86d71c428be-maas-api-xks-ca\") pod \"e869a01e-b2dc-414c-949b-f86d71c428be\" (UID: \"e869a01e-b2dc-414c-949b-f86d71c428be\") "
Nov 02 18:12:10.567501 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.567208    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e869a01e-b2dc-414c-949b-f86d71c428be-maas-api-openshift-ca" pod "e869a01e-b2dc-414c-949b-f86d71c428be" (UID: "e869a01e-b2dc-414c-949b-f86d71c428be"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:10.567501 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.567472    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/e869a01e-b2dc-414c-949b-f86d71c428be-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:10.569338 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.569313    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e869a01e-b2dc-414c-949b-f86d71c428be-kube-api-access-p6vwh" pod "e869a01e-b2dc-414c-949b-f86d71c428be" (UID: "e869a01e-b2dc-414c-949b-f86d71c428be"). InnerVolumeSpecName "kube-api-access-p6vwh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:10.569793 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.569777    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/e869a01e-b2dc-414c-949b-f86d71c428be-maas-api-xks-ca" pod "e869a01e-b2dc-414c-949b-f86d71c428be" (UID: "e869a01e-b2dc-414c-949b-f86d71c428be"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:10.670888 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.669846    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-p6vwh\" (UniqueName: \"kubernetes.io/projected/e869a01e-b2dc-414c-949b-f86d71c428be-kube-api-access-p6vwh\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:10.670888 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.669885    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/e869a01e-b2dc-414c-949b-f86d71c428be-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:10.787157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.787118    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:12:10.871896 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.871867    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"]
Nov 02 18:12:10.872981 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:12:10.872956    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8045e931_2116_422f_806e_f18e3b757a0c.slice/crio-ec3fcf9232b98bef116faa03eef1fa9479366b14e49ab08d04448513dc4ddb6c WatchSource:0}: Error finding container ec3fcf9232b98bef116faa03eef1fa9479366b14e49ab08d04448513dc4ddb6c: Status 404 returned error can't find the container with id ec3fcf9232b98bef116faa03eef1fa9479366b14e49ab08d04448513dc4ddb6c
Nov 02 18:12:10.992564 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:10.992529    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:12:11.185222 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:11.185192    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp" event={"ID":"e869a01e-b2dc-414c-949b-f86d71c428be","Type":"ContainerDied","Data":"53b5afd84af0711011c0219889632712acbf19f3b7e46215a8f31b72b1783c72"}
Nov 02 18:12:11.185222 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:11.185212    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"
Nov 02 18:12:11.185421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:11.185247    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53b5afd84af0711011c0219889632712acbf19f3b7e46215a8f31b72b1783c72"
Nov 02 18:12:11.186669 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:11.186643    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv" event={"ID":"8045e931-2116-422f-806e-f18e3b757a0c","Type":"ContainerStarted","Data":"ec3fcf9232b98bef116faa03eef1fa9479366b14e49ab08d04448513dc4ddb6c"}
Nov 02 18:12:11.188315 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:11.188292    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:12:11.743243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:11.743210    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:12.200223 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:12.200192    4456 generic.go:356] "Generic (PLEG): container finished" podID="8045e931-2116-422f-806e-f18e3b757a0c" containerID="92cdf4068fea46ac976b13d8439a63750fbece3874b0e28b3203b8dac5d5fec8" exitCode=0
Nov 02 18:12:12.200383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:12.200286    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv" event={"ID":"8045e931-2116-422f-806e-f18e3b757a0c","Type":"ContainerDied","Data":"92cdf4068fea46ac976b13d8439a63750fbece3874b0e28b3203b8dac5d5fec8"}
Nov 02 18:12:12.834845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:12.834797    4456 scope.go:122] "RemoveContainer" containerID="ddbb7b95206d8fc7e518db1afa3373123bf45a884a122a3a8a12139090ad5df4"
Nov 02 18:12:12.846364 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:12.846345    4456 scope.go:122] "RemoveContainer" containerID="0afcedd1deec4a2d5cae278ee100bd19f987c5e216a88b6ff7615e95ad5c8ed5"
Nov 02 18:12:12.857430 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:12.857406    4456 scope.go:122] "RemoveContainer" containerID="eb0f22e0b2050b9356e1474bc817e81e1dc7dc4d3c6640daa55886540961bf5c"
Nov 02 18:12:12.867064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:12.867048    4456 scope.go:122] "RemoveContainer" containerID="0dcde13a6a52ff4eff6e292ac044743343cbb0afa2e3f9446f8189685a9bd665"
Nov 02 18:12:13.090596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:13.090571    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:12:13.368972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:13.368949    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"
Nov 02 18:12:13.413754 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:13.413719    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:13.507040 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:13.507003    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8045e931-2116-422f-806e-f18e3b757a0c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8045e931-2116-422f-806e-f18e3b757a0c-maas-api-xks-ca\") pod \"8045e931-2116-422f-806e-f18e3b757a0c\" (UID: \"8045e931-2116-422f-806e-f18e3b757a0c\") "
Nov 02 18:12:13.507248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:13.507230    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8045e931-2116-422f-806e-f18e3b757a0c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8045e931-2116-422f-806e-f18e3b757a0c-maas-api-openshift-ca\") pod \"8045e931-2116-422f-806e-f18e3b757a0c\" (UID: \"8045e931-2116-422f-806e-f18e3b757a0c\") "
Nov 02 18:12:13.507309 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:13.507298    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8045e931-2116-422f-806e-f18e3b757a0c-kube-api-access-g87vd\" (UniqueName: \"kubernetes.io/projected/8045e931-2116-422f-806e-f18e3b757a0c-kube-api-access-g87vd\") pod \"8045e931-2116-422f-806e-f18e3b757a0c\" (UID: \"8045e931-2116-422f-806e-f18e3b757a0c\") "
Nov 02 18:12:13.507572 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:13.507550    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8045e931-2116-422f-806e-f18e3b757a0c-maas-api-openshift-ca" pod "8045e931-2116-422f-806e-f18e3b757a0c" (UID: "8045e931-2116-422f-806e-f18e3b757a0c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:13.507762 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:13.507741    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8045e931-2116-422f-806e-f18e3b757a0c-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:13.509319 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:13.509288    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8045e931-2116-422f-806e-f18e3b757a0c-maas-api-xks-ca" pod "8045e931-2116-422f-806e-f18e3b757a0c" (UID: "8045e931-2116-422f-806e-f18e3b757a0c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:13.509736 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:13.509715    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8045e931-2116-422f-806e-f18e3b757a0c-kube-api-access-g87vd" pod "8045e931-2116-422f-806e-f18e3b757a0c" (UID: "8045e931-2116-422f-806e-f18e3b757a0c"). InnerVolumeSpecName "kube-api-access-g87vd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:13.609401 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:13.609322    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8045e931-2116-422f-806e-f18e3b757a0c-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:13.609401 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:13.609353    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g87vd\" (UniqueName: \"kubernetes.io/projected/8045e931-2116-422f-806e-f18e3b757a0c-kube-api-access-g87vd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:14.219929 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:14.219894    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"
Nov 02 18:12:14.220324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:14.219938    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv" event={"ID":"8045e931-2116-422f-806e-f18e3b757a0c","Type":"ContainerDied","Data":"ec3fcf9232b98bef116faa03eef1fa9479366b14e49ab08d04448513dc4ddb6c"}
Nov 02 18:12:14.220324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:14.219996    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ec3fcf9232b98bef116faa03eef1fa9479366b14e49ab08d04448513dc4ddb6c"
Nov 02 18:12:14.714100 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:14.714032    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" podUID="e4c1132f-05b5-4664-84a2-ece57727b12a" containerName="payload-pre-processing" containerID="cri-o://a3c609da08ba303e35942c668188b131f4c241ad4c0f9869076f8590a95a0a00" gracePeriod=25
Nov 02 18:12:15.007206 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.007179    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 18:12:15.110982 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.110948    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:12:15.128996 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.128964    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e4c1132f-05b5-4664-84a2-ece57727b12a-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c1132f-05b5-4664-84a2-ece57727b12a-plugins-config-volume\") pod \"e4c1132f-05b5-4664-84a2-ece57727b12a\" (UID: \"e4c1132f-05b5-4664-84a2-ece57727b12a\") "
Nov 02 18:12:15.129131 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.129081    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e4c1132f-05b5-4664-84a2-ece57727b12a-kube-api-access-d5jvw\" (UniqueName: \"kubernetes.io/projected/e4c1132f-05b5-4664-84a2-ece57727b12a-kube-api-access-d5jvw\") pod \"e4c1132f-05b5-4664-84a2-ece57727b12a\" (UID: \"e4c1132f-05b5-4664-84a2-ece57727b12a\") "
Nov 02 18:12:15.129371 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.129349    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e4c1132f-05b5-4664-84a2-ece57727b12a-plugins-config-volume" pod "e4c1132f-05b5-4664-84a2-ece57727b12a" (UID: "e4c1132f-05b5-4664-84a2-ece57727b12a"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:15.129561 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.129535    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e4c1132f-05b5-4664-84a2-ece57727b12a-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:15.131668 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.131639    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e4c1132f-05b5-4664-84a2-ece57727b12a-kube-api-access-d5jvw" pod "e4c1132f-05b5-4664-84a2-ece57727b12a" (UID: "e4c1132f-05b5-4664-84a2-ece57727b12a"). InnerVolumeSpecName "kube-api-access-d5jvw". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:15.231246 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.231153    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d5jvw\" (UniqueName: \"kubernetes.io/projected/e4c1132f-05b5-4664-84a2-ece57727b12a-kube-api-access-d5jvw\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:15.233523 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.233492    4456 generic.go:356] "Generic (PLEG): container finished" podID="e4c1132f-05b5-4664-84a2-ece57727b12a" containerID="a3c609da08ba303e35942c668188b131f4c241ad4c0f9869076f8590a95a0a00" exitCode=0
Nov 02 18:12:15.233643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.233545    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"
Nov 02 18:12:15.233643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.233581    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" event={"ID":"e4c1132f-05b5-4664-84a2-ece57727b12a","Type":"ContainerDied","Data":"a3c609da08ba303e35942c668188b131f4c241ad4c0f9869076f8590a95a0a00"}
Nov 02 18:12:15.233643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.233635    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8" event={"ID":"e4c1132f-05b5-4664-84a2-ece57727b12a","Type":"ContainerDied","Data":"44e06805a9771401184c6e3698e3e96b09d126e5304a78e131abf3ee46746816"}
Nov 02 18:12:15.233756 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.233659    4456 scope.go:122] "RemoveContainer" containerID="a3c609da08ba303e35942c668188b131f4c241ad4c0f9869076f8590a95a0a00"
Nov 02 18:12:15.246065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.246042    4456 scope.go:122] "RemoveContainer" containerID="a3c609da08ba303e35942c668188b131f4c241ad4c0f9869076f8590a95a0a00"
Nov 02 18:12:15.246370 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:15.246349    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3c609da08ba303e35942c668188b131f4c241ad4c0f9869076f8590a95a0a00\": container with ID starting with a3c609da08ba303e35942c668188b131f4c241ad4c0f9869076f8590a95a0a00 not found: ID does not exist" containerID="a3c609da08ba303e35942c668188b131f4c241ad4c0f9869076f8590a95a0a00"
Nov 02 18:12:15.246422 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.246378    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3c609da08ba303e35942c668188b131f4c241ad4c0f9869076f8590a95a0a00"} err="failed to get container status \"a3c609da08ba303e35942c668188b131f4c241ad4c0f9869076f8590a95a0a00\": rpc error: code = NotFound desc = could not find container \"a3c609da08ba303e35942c668188b131f4c241ad4c0f9869076f8590a95a0a00\": container with ID starting with a3c609da08ba303e35942c668188b131f4c241ad4c0f9869076f8590a95a0a00 not found: ID does not exist"
Nov 02 18:12:15.290873 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.290840    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"]
Nov 02 18:12:15.294414 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.294391    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w5-504de7-5bbc9b5879-9nhl8"]
Nov 02 18:12:15.320238 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.320175    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:15.615252 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:15.615171    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e4c1132f-05b5-4664-84a2-ece57727b12a" path="/var/lib/kubelet/pods/e4c1132f-05b5-4664-84a2-ece57727b12a/volumes"
Nov 02 18:12:16.067314 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:16.067277    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:16.166122 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:16.166089    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:12:16.225833 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:16.225788    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:16.370799 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:16.370722    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:16.802053 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:16.802022    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:16.918497 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:16.918462    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"]
Nov 02 18:12:16.953081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:16.953051    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"]
Nov 02 18:12:17.091498 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:17.091413    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:17.237420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:17.237388    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:17.909809 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:17.909774    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:18.069292 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:18.069257    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:18.961863 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:18.961823    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:20.719774 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:20.719736    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"]
Nov 02 18:12:20.837751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:20.837710    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:20.983177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:20.983076    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:21.919723 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:21.919680    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerName="payload-processing" containerID="cri-o://865d4366efe4897f0fbc45fe62ca1b7caf82220eebf3378b67df5abe034fc0ed" gracePeriod=25
Nov 02 18:12:21.953358 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:21.953320    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5" podUID="02a2dbe6-baeb-40c4-be0f-835239be331f" containerName="payload-pre-processing" containerID="cri-o://ac1aa45ece34b79f424daed8874251c74338193ccb0a28d4a390b8a7bb7763f2" gracePeriod=25
Nov 02 18:12:22.232496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.232143    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 18:12:22.269399 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.269377    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"
Nov 02 18:12:22.286270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.286215    4456 generic.go:356] "Generic (PLEG): container finished" podID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerID="865d4366efe4897f0fbc45fe62ca1b7caf82220eebf3378b67df5abe034fc0ed" exitCode=0
Nov 02 18:12:22.286468 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.286443    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" event={"ID":"cc682b81-5f9e-4cd1-98ae-d32e89878ea3","Type":"ContainerDied","Data":"865d4366efe4897f0fbc45fe62ca1b7caf82220eebf3378b67df5abe034fc0ed"}
Nov 02 18:12:22.286592 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.286501    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb" event={"ID":"cc682b81-5f9e-4cd1-98ae-d32e89878ea3","Type":"ContainerDied","Data":"385fb2039950420031b2c48061b1d32b4c8aa2739ace21371a79a83141090739"}
Nov 02 18:12:22.286592 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.286532    4456 scope.go:122] "RemoveContainer" containerID="865d4366efe4897f0fbc45fe62ca1b7caf82220eebf3378b67df5abe034fc0ed"
Nov 02 18:12:22.286696 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.286678    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"
Nov 02 18:12:22.289176 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.289151    4456 generic.go:356] "Generic (PLEG): container finished" podID="02a2dbe6-baeb-40c4-be0f-835239be331f" containerID="ac1aa45ece34b79f424daed8874251c74338193ccb0a28d4a390b8a7bb7763f2" exitCode=0
Nov 02 18:12:22.289309 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.289200    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"
Nov 02 18:12:22.289309 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.289243    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5" event={"ID":"02a2dbe6-baeb-40c4-be0f-835239be331f","Type":"ContainerDied","Data":"ac1aa45ece34b79f424daed8874251c74338193ccb0a28d4a390b8a7bb7763f2"}
Nov 02 18:12:22.289309 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.289291    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5" event={"ID":"02a2dbe6-baeb-40c4-be0f-835239be331f","Type":"ContainerDied","Data":"099e22181ceccbf59f14ab74c24153a9fbe4ff720a3ff4ff9e26d14fe45f9fc4"}
Nov 02 18:12:22.302105 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.302079    4456 scope.go:122] "RemoveContainer" containerID="865d4366efe4897f0fbc45fe62ca1b7caf82220eebf3378b67df5abe034fc0ed"
Nov 02 18:12:22.302506 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:22.302472    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"865d4366efe4897f0fbc45fe62ca1b7caf82220eebf3378b67df5abe034fc0ed\": container with ID starting with 865d4366efe4897f0fbc45fe62ca1b7caf82220eebf3378b67df5abe034fc0ed not found: ID does not exist" containerID="865d4366efe4897f0fbc45fe62ca1b7caf82220eebf3378b67df5abe034fc0ed"
Nov 02 18:12:22.302615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.302515    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"865d4366efe4897f0fbc45fe62ca1b7caf82220eebf3378b67df5abe034fc0ed"} err="failed to get container status \"865d4366efe4897f0fbc45fe62ca1b7caf82220eebf3378b67df5abe034fc0ed\": rpc error: code = NotFound desc = could not find container \"865d4366efe4897f0fbc45fe62ca1b7caf82220eebf3378b67df5abe034fc0ed\": container with ID starting with 865d4366efe4897f0fbc45fe62ca1b7caf82220eebf3378b67df5abe034fc0ed not found: ID does not exist"
Nov 02 18:12:22.302615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.302538    4456 scope.go:122] "RemoveContainer" containerID="ac1aa45ece34b79f424daed8874251c74338193ccb0a28d4a390b8a7bb7763f2"
Nov 02 18:12:22.316500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.316475    4456 scope.go:122] "RemoveContainer" containerID="ac1aa45ece34b79f424daed8874251c74338193ccb0a28d4a390b8a7bb7763f2"
Nov 02 18:12:22.317040 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:22.316859    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ac1aa45ece34b79f424daed8874251c74338193ccb0a28d4a390b8a7bb7763f2\": container with ID starting with ac1aa45ece34b79f424daed8874251c74338193ccb0a28d4a390b8a7bb7763f2 not found: ID does not exist" containerID="ac1aa45ece34b79f424daed8874251c74338193ccb0a28d4a390b8a7bb7763f2"
Nov 02 18:12:22.317040 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.316898    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ac1aa45ece34b79f424daed8874251c74338193ccb0a28d4a390b8a7bb7763f2"} err="failed to get container status \"ac1aa45ece34b79f424daed8874251c74338193ccb0a28d4a390b8a7bb7763f2\": rpc error: code = NotFound desc = could not find container \"ac1aa45ece34b79f424daed8874251c74338193ccb0a28d4a390b8a7bb7763f2\": container with ID starting with ac1aa45ece34b79f424daed8874251c74338193ccb0a28d4a390b8a7bb7763f2 not found: ID does not exist"
Nov 02 18:12:22.345371 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.345339    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/02a2dbe6-baeb-40c4-be0f-835239be331f-kube-api-access-ccrc9\" (UniqueName: \"kubernetes.io/projected/02a2dbe6-baeb-40c4-be0f-835239be331f-kube-api-access-ccrc9\") pod \"02a2dbe6-baeb-40c4-be0f-835239be331f\" (UID: \"02a2dbe6-baeb-40c4-be0f-835239be331f\") "
Nov 02 18:12:22.345512 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.345453    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc682b81-5f9e-4cd1-98ae-d32e89878ea3-kube-api-access-cssvv\" (UniqueName: \"kubernetes.io/projected/cc682b81-5f9e-4cd1-98ae-d32e89878ea3-kube-api-access-cssvv\") pod \"cc682b81-5f9e-4cd1-98ae-d32e89878ea3\" (UID: \"cc682b81-5f9e-4cd1-98ae-d32e89878ea3\") "
Nov 02 18:12:22.345512 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.345496    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/02a2dbe6-baeb-40c4-be0f-835239be331f-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a2dbe6-baeb-40c4-be0f-835239be331f-plugins-config-volume\") pod \"02a2dbe6-baeb-40c4-be0f-835239be331f\" (UID: \"02a2dbe6-baeb-40c4-be0f-835239be331f\") "
Nov 02 18:12:22.345634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.345594    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc682b81-5f9e-4cd1-98ae-d32e89878ea3-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc682b81-5f9e-4cd1-98ae-d32e89878ea3-plugins-config-volume\") pod \"cc682b81-5f9e-4cd1-98ae-d32e89878ea3\" (UID: \"cc682b81-5f9e-4cd1-98ae-d32e89878ea3\") "
Nov 02 18:12:22.345991 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.345961    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/02a2dbe6-baeb-40c4-be0f-835239be331f-plugins-config-volume" pod "02a2dbe6-baeb-40c4-be0f-835239be331f" (UID: "02a2dbe6-baeb-40c4-be0f-835239be331f"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:22.346271 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.346030    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc682b81-5f9e-4cd1-98ae-d32e89878ea3-plugins-config-volume" pod "cc682b81-5f9e-4cd1-98ae-d32e89878ea3" (UID: "cc682b81-5f9e-4cd1-98ae-d32e89878ea3"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:22.347702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.347680    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc682b81-5f9e-4cd1-98ae-d32e89878ea3-kube-api-access-cssvv" pod "cc682b81-5f9e-4cd1-98ae-d32e89878ea3" (UID: "cc682b81-5f9e-4cd1-98ae-d32e89878ea3"). InnerVolumeSpecName "kube-api-access-cssvv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:22.348105 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.348088    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/02a2dbe6-baeb-40c4-be0f-835239be331f-kube-api-access-ccrc9" pod "02a2dbe6-baeb-40c4-be0f-835239be331f" (UID: "02a2dbe6-baeb-40c4-be0f-835239be331f"). InnerVolumeSpecName "kube-api-access-ccrc9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:22.447443 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.447353    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cssvv\" (UniqueName: \"kubernetes.io/projected/cc682b81-5f9e-4cd1-98ae-d32e89878ea3-kube-api-access-cssvv\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:22.447443 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.447392    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/02a2dbe6-baeb-40c4-be0f-835239be331f-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:22.447443 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.447402    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/cc682b81-5f9e-4cd1-98ae-d32e89878ea3-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:22.447443 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.447411    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ccrc9\" (UniqueName: \"kubernetes.io/projected/02a2dbe6-baeb-40c4-be0f-835239be331f-kube-api-access-ccrc9\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:22.646297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.646259    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"]
Nov 02 18:12:22.658494 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.658451    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-a-w4-f59572-7cc4b56f45-tl4jb"]
Nov 02 18:12:22.659364 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.659335    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:22.696477 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.696435    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"]
Nov 02 18:12:22.703121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:22.703047    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w4-f59572-86467c468c-t5wf5"]
Nov 02 18:12:23.090179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:23.090092    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:12:23.615905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:23.615873    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="02a2dbe6-baeb-40c4-be0f-835239be331f" path="/var/lib/kubelet/pods/02a2dbe6-baeb-40c4-be0f-835239be331f/volumes"
Nov 02 18:12:23.616363 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:23.616348    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" path="/var/lib/kubelet/pods/cc682b81-5f9e-4cd1-98ae-d32e89878ea3/volumes"
Nov 02 18:12:24.239781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:24.239741    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:24.700807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:24.699169    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:24.848036 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:24.847991    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:25.110412 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:25.110330    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:12:25.118430 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:25.118397    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"]
Nov 02 18:12:25.129597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:25.129563    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w6-b91583-nt4nh"]
Nov 02 18:12:25.191865 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:25.191324    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:25.616189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:25.616159    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f54bd962-e719-41c4-9c28-56460e46d0f6" path="/var/lib/kubelet/pods/f54bd962-e719-41c4-9c28-56460e46d0f6/volumes"
Nov 02 18:12:25.720448 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:25.720405    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2" podUID="49238ac8-57fa-412a-88bd-681364a0b455" containerName="payload-pre-processing" containerID="cri-o://1078c01c652fa0ea5d83259df4fe78a4cf54da34d16748e6c4e9933597f21ac0" gracePeriod=25
Nov 02 18:12:25.995277 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:25.995250    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"
Nov 02 18:12:26.108695 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.108657    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/49238ac8-57fa-412a-88bd-681364a0b455-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49238ac8-57fa-412a-88bd-681364a0b455-plugins-config-volume\") pod \"49238ac8-57fa-412a-88bd-681364a0b455\" (UID: \"49238ac8-57fa-412a-88bd-681364a0b455\") "
Nov 02 18:12:26.108883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.108768    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/49238ac8-57fa-412a-88bd-681364a0b455-kube-api-access-8srp8\" (UniqueName: \"kubernetes.io/projected/49238ac8-57fa-412a-88bd-681364a0b455-kube-api-access-8srp8\") pod \"49238ac8-57fa-412a-88bd-681364a0b455\" (UID: \"49238ac8-57fa-412a-88bd-681364a0b455\") "
Nov 02 18:12:26.109229 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.109139    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/49238ac8-57fa-412a-88bd-681364a0b455-plugins-config-volume" pod "49238ac8-57fa-412a-88bd-681364a0b455" (UID: "49238ac8-57fa-412a-88bd-681364a0b455"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:26.109421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.109402    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/49238ac8-57fa-412a-88bd-681364a0b455-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:26.111846 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.111784    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/49238ac8-57fa-412a-88bd-681364a0b455-kube-api-access-8srp8" pod "49238ac8-57fa-412a-88bd-681364a0b455" (UID: "49238ac8-57fa-412a-88bd-681364a0b455"). InnerVolumeSpecName "kube-api-access-8srp8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:26.175018 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.174934    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:26.210718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.210672    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8srp8\" (UniqueName: \"kubernetes.io/projected/49238ac8-57fa-412a-88bd-681364a0b455-kube-api-access-8srp8\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:26.325050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.325012    4456 generic.go:356] "Generic (PLEG): container finished" podID="49238ac8-57fa-412a-88bd-681364a0b455" containerID="1078c01c652fa0ea5d83259df4fe78a4cf54da34d16748e6c4e9933597f21ac0" exitCode=0
Nov 02 18:12:26.325248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.325221    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2" event={"ID":"49238ac8-57fa-412a-88bd-681364a0b455","Type":"ContainerDied","Data":"1078c01c652fa0ea5d83259df4fe78a4cf54da34d16748e6c4e9933597f21ac0"}
Nov 02 18:12:26.325312 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.325267    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2" event={"ID":"49238ac8-57fa-412a-88bd-681364a0b455","Type":"ContainerDied","Data":"6238a8826930486e5da032e1da913157c2129d65eff9738f90f2276137fb9f80"}
Nov 02 18:12:26.325312 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.325296    4456 scope.go:122] "RemoveContainer" containerID="1078c01c652fa0ea5d83259df4fe78a4cf54da34d16748e6c4e9933597f21ac0"
Nov 02 18:12:26.325433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.325419    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"
Nov 02 18:12:26.340458 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.340429    4456 scope.go:122] "RemoveContainer" containerID="1078c01c652fa0ea5d83259df4fe78a4cf54da34d16748e6c4e9933597f21ac0"
Nov 02 18:12:26.340789 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:26.340763    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1078c01c652fa0ea5d83259df4fe78a4cf54da34d16748e6c4e9933597f21ac0\": container with ID starting with 1078c01c652fa0ea5d83259df4fe78a4cf54da34d16748e6c4e9933597f21ac0 not found: ID does not exist" containerID="1078c01c652fa0ea5d83259df4fe78a4cf54da34d16748e6c4e9933597f21ac0"
Nov 02 18:12:26.340916 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.340802    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1078c01c652fa0ea5d83259df4fe78a4cf54da34d16748e6c4e9933597f21ac0"} err="failed to get container status \"1078c01c652fa0ea5d83259df4fe78a4cf54da34d16748e6c4e9933597f21ac0\": rpc error: code = NotFound desc = could not find container \"1078c01c652fa0ea5d83259df4fe78a4cf54da34d16748e6c4e9933597f21ac0\": container with ID starting with 1078c01c652fa0ea5d83259df4fe78a4cf54da34d16748e6c4e9933597f21ac0 not found: ID does not exist"
Nov 02 18:12:26.383137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.383097    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"]
Nov 02 18:12:26.408163 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.408125    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-shared-a-w3-557feb-57d7976c4c-x2fz2"]
Nov 02 18:12:26.595627 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:26.595576    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:27.054077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:27.054038    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:27.613761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:27.613720    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="49238ac8-57fa-412a-88bd-681364a0b455" path="/var/lib/kubelet/pods/49238ac8-57fa-412a-88bd-681364a0b455/volumes"
Nov 02 18:12:28.572045 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:28.572013    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:28.985907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:28.985877    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:29.147319 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:29.147274    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:29.177318 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:29.177289    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"]
Nov 02 18:12:29.223154 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:29.223123    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-96768b-g6rcr"]
Nov 02 18:12:29.468164 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:29.468129    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:29.611726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:29.611685    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:29.618607 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:29.618578    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d8ec64b7-c3f5-4493-8657-07fe497a12aa" path="/var/lib/kubelet/pods/d8ec64b7-c3f5-4493-8657-07fe497a12aa/volumes"
Nov 02 18:12:29.657264 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:29.657232    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"]
Nov 02 18:12:29.660688 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:29.660665    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-a29999-vmshb"]
Nov 02 18:12:29.891835 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:29.888994    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"
Nov 02 18:12:29.920460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:29.919885    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"
Nov 02 18:12:29.985302 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:29.985222    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:30.073068 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.073039    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b5d45c38-1cef-4aa9-a519-98aed00e8727-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d45c38-1cef-4aa9-a519-98aed00e8727-plugins-config-volume\") pod \"b5d45c38-1cef-4aa9-a519-98aed00e8727\" (UID: \"b5d45c38-1cef-4aa9-a519-98aed00e8727\") "
Nov 02 18:12:30.073235 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.073088    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b5d45c38-1cef-4aa9-a519-98aed00e8727-kube-api-access-2d9tp\" (UniqueName: \"kubernetes.io/projected/b5d45c38-1cef-4aa9-a519-98aed00e8727-kube-api-access-2d9tp\") pod \"b5d45c38-1cef-4aa9-a519-98aed00e8727\" (UID: \"b5d45c38-1cef-4aa9-a519-98aed00e8727\") "
Nov 02 18:12:30.073235 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.073189    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18a32a7b-66fa-4f34-ab36-2beb0169e430-kube-api-access-dvhdh\" (UniqueName: \"kubernetes.io/projected/18a32a7b-66fa-4f34-ab36-2beb0169e430-kube-api-access-dvhdh\") pod \"18a32a7b-66fa-4f34-ab36-2beb0169e430\" (UID: \"18a32a7b-66fa-4f34-ab36-2beb0169e430\") "
Nov 02 18:12:30.073235 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.073227    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18a32a7b-66fa-4f34-ab36-2beb0169e430-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a32a7b-66fa-4f34-ab36-2beb0169e430-plugins-config-volume\") pod \"18a32a7b-66fa-4f34-ab36-2beb0169e430\" (UID: \"18a32a7b-66fa-4f34-ab36-2beb0169e430\") "
Nov 02 18:12:30.073567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.073489    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b5d45c38-1cef-4aa9-a519-98aed00e8727-plugins-config-volume" pod "b5d45c38-1cef-4aa9-a519-98aed00e8727" (UID: "b5d45c38-1cef-4aa9-a519-98aed00e8727"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:30.073759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.073739    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18a32a7b-66fa-4f34-ab36-2beb0169e430-plugins-config-volume" pod "18a32a7b-66fa-4f34-ab36-2beb0169e430" (UID: "18a32a7b-66fa-4f34-ab36-2beb0169e430"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:30.073964 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.073942    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b5d45c38-1cef-4aa9-a519-98aed00e8727-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:30.074046 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.073967    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/18a32a7b-66fa-4f34-ab36-2beb0169e430-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:30.075509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.075490    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b5d45c38-1cef-4aa9-a519-98aed00e8727-kube-api-access-2d9tp" pod "b5d45c38-1cef-4aa9-a519-98aed00e8727" (UID: "b5d45c38-1cef-4aa9-a519-98aed00e8727"). InnerVolumeSpecName "kube-api-access-2d9tp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:30.075558 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.075542    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18a32a7b-66fa-4f34-ab36-2beb0169e430-kube-api-access-dvhdh" pod "18a32a7b-66fa-4f34-ab36-2beb0169e430" (UID: "18a32a7b-66fa-4f34-ab36-2beb0169e430"). InnerVolumeSpecName "kube-api-access-dvhdh". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:30.135081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.135043    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"]
Nov 02 18:12:30.137999 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.137971    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-inf-a-bb6d2e-vwlnr"]
Nov 02 18:12:30.142775 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.142741    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:30.174668 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.174631    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2d9tp\" (UniqueName: \"kubernetes.io/projected/b5d45c38-1cef-4aa9-a519-98aed00e8727-kube-api-access-2d9tp\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:30.174668 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.174663    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-dvhdh\" (UniqueName: \"kubernetes.io/projected/18a32a7b-66fa-4f34-ab36-2beb0169e430-kube-api-access-dvhdh\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:30.371381 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.371298    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"]
Nov 02 18:12:30.371894 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.371875    4456 cpu_manager.go:414] "containerMap: removing container" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerName="payload-processing"
Nov 02 18:12:30.372002 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.371895    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerName="payload-processing"
Nov 02 18:12:30.372002 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.371914    4456 cpu_manager.go:414] "containerMap: removing container" podUID="8045e931-2116-422f-806e-f18e3b757a0c" containerName="revoke-keys"
Nov 02 18:12:30.372002 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.371922    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8045e931-2116-422f-806e-f18e3b757a0c" containerName="revoke-keys"
Nov 02 18:12:30.372002 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.371949    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e4c1132f-05b5-4664-84a2-ece57727b12a" containerName="payload-pre-processing"
Nov 02 18:12:30.372002 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.371958    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e4c1132f-05b5-4664-84a2-ece57727b12a" containerName="payload-pre-processing"
Nov 02 18:12:30.372002 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.371973    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e869a01e-b2dc-414c-949b-f86d71c428be" containerName="revoke-keys"
Nov 02 18:12:30.372002 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.371980    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e869a01e-b2dc-414c-949b-f86d71c428be" containerName="revoke-keys"
Nov 02 18:12:30.372327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372021    4456 cpu_manager.go:414] "containerMap: removing container" podUID="02a2dbe6-baeb-40c4-be0f-835239be331f" containerName="payload-pre-processing"
Nov 02 18:12:30.372327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372029    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="02a2dbe6-baeb-40c4-be0f-835239be331f" containerName="payload-pre-processing"
Nov 02 18:12:30.372327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372041    4456 cpu_manager.go:414] "containerMap: removing container" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing"
Nov 02 18:12:30.372327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372049    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing"
Nov 02 18:12:30.372327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372065    4456 cpu_manager.go:414] "containerMap: removing container" podUID="49238ac8-57fa-412a-88bd-681364a0b455" containerName="payload-pre-processing"
Nov 02 18:12:30.372327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372072    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="49238ac8-57fa-412a-88bd-681364a0b455" containerName="payload-pre-processing"
Nov 02 18:12:30.372327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372085    4456 cpu_manager.go:414] "containerMap: removing container" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing"
Nov 02 18:12:30.372327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372092    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing"
Nov 02 18:12:30.372327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372222    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e4c1132f-05b5-4664-84a2-ece57727b12a" containerName="payload-pre-processing"
Nov 02 18:12:30.372327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372236    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e869a01e-b2dc-414c-949b-f86d71c428be" containerName="revoke-keys"
Nov 02 18:12:30.372327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372253    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="02a2dbe6-baeb-40c4-be0f-835239be331f" containerName="payload-pre-processing"
Nov 02 18:12:30.372794 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372261    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerName="payload-processing"
Nov 02 18:12:30.372794 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372271    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="49238ac8-57fa-412a-88bd-681364a0b455" containerName="payload-pre-processing"
Nov 02 18:12:30.372794 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372282    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerName="payload-pre-processing"
Nov 02 18:12:30.372794 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372293    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc682b81-5f9e-4cd1-98ae-d32e89878ea3" containerName="payload-processing"
Nov 02 18:12:30.372794 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.372304    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8045e931-2116-422f-806e-f18e3b757a0c" containerName="revoke-keys"
Nov 02 18:12:30.376067 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.376044    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"
Nov 02 18:12:30.390209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.390178    4456 generic.go:356] "Generic (PLEG): container finished" podID="18a32a7b-66fa-4f34-ab36-2beb0169e430" containerID="65dc1c7055ca0c593790169b343803715577f608817bf5ba500657f16af44d0b" exitCode=0
Nov 02 18:12:30.390339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.390234    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"
Nov 02 18:12:30.390339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.390257    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" event={"ID":"18a32a7b-66fa-4f34-ab36-2beb0169e430","Type":"ContainerDied","Data":"65dc1c7055ca0c593790169b343803715577f608817bf5ba500657f16af44d0b"}
Nov 02 18:12:30.390339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.390320    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4" event={"ID":"18a32a7b-66fa-4f34-ab36-2beb0169e430","Type":"ContainerDied","Data":"695b8b70dea01fa843ea28ef2b7e9e8b64d8246dc2edfad876806e3044f16058"}
Nov 02 18:12:30.390565 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.390352    4456 scope.go:122] "RemoveContainer" containerID="65dc1c7055ca0c593790169b343803715577f608817bf5ba500657f16af44d0b"
Nov 02 18:12:30.392700 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.392379    4456 generic.go:356] "Generic (PLEG): container finished" podID="b5d45c38-1cef-4aa9-a519-98aed00e8727" containerID="fb0803029471e70042acd1e45626801c9ff68565bd8c57f88a3acdbb82e2b010" exitCode=0
Nov 02 18:12:30.392700 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.392586    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" event={"ID":"b5d45c38-1cef-4aa9-a519-98aed00e8727","Type":"ContainerDied","Data":"fb0803029471e70042acd1e45626801c9ff68565bd8c57f88a3acdbb82e2b010"}
Nov 02 18:12:30.392700 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.392624    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz" event={"ID":"b5d45c38-1cef-4aa9-a519-98aed00e8727","Type":"ContainerDied","Data":"0183eddcebc7b4d46fa8e5bbf2be7bd6ff098f83a45e201f0cb7a81c25790914"}
Nov 02 18:12:30.392700 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.392688    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"
Nov 02 18:12:30.411037 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.410553    4456 scope.go:122] "RemoveContainer" containerID="65dc1c7055ca0c593790169b343803715577f608817bf5ba500657f16af44d0b"
Nov 02 18:12:30.417660 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.417625    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"]
Nov 02 18:12:30.418335 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:30.418085    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"65dc1c7055ca0c593790169b343803715577f608817bf5ba500657f16af44d0b\": container with ID starting with 65dc1c7055ca0c593790169b343803715577f608817bf5ba500657f16af44d0b not found: ID does not exist" containerID="65dc1c7055ca0c593790169b343803715577f608817bf5ba500657f16af44d0b"
Nov 02 18:12:30.418335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.418179    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"65dc1c7055ca0c593790169b343803715577f608817bf5ba500657f16af44d0b"} err="failed to get container status \"65dc1c7055ca0c593790169b343803715577f608817bf5ba500657f16af44d0b\": rpc error: code = NotFound desc = could not find container \"65dc1c7055ca0c593790169b343803715577f608817bf5ba500657f16af44d0b\": container with ID starting with 65dc1c7055ca0c593790169b343803715577f608817bf5ba500657f16af44d0b not found: ID does not exist"
Nov 02 18:12:30.418335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.418207    4456 scope.go:122] "RemoveContainer" containerID="fb0803029471e70042acd1e45626801c9ff68565bd8c57f88a3acdbb82e2b010"
Nov 02 18:12:30.428405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.420766    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"]
Nov 02 18:12:30.449835 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.444955    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:12:30.449835 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.446908    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:12:30.450266 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.450241    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"
Nov 02 18:12:30.469116 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.469092    4456 scope.go:122] "RemoveContainer" containerID="fb0803029471e70042acd1e45626801c9ff68565bd8c57f88a3acdbb82e2b010"
Nov 02 18:12:30.470104 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:30.470071    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"fb0803029471e70042acd1e45626801c9ff68565bd8c57f88a3acdbb82e2b010\": container with ID starting with fb0803029471e70042acd1e45626801c9ff68565bd8c57f88a3acdbb82e2b010 not found: ID does not exist" containerID="fb0803029471e70042acd1e45626801c9ff68565bd8c57f88a3acdbb82e2b010"
Nov 02 18:12:30.470230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.470112    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"fb0803029471e70042acd1e45626801c9ff68565bd8c57f88a3acdbb82e2b010"} err="failed to get container status \"fb0803029471e70042acd1e45626801c9ff68565bd8c57f88a3acdbb82e2b010\": rpc error: code = NotFound desc = could not find container \"fb0803029471e70042acd1e45626801c9ff68565bd8c57f88a3acdbb82e2b010\": container with ID starting with fb0803029471e70042acd1e45626801c9ff68565bd8c57f88a3acdbb82e2b010 not found: ID does not exist"
Nov 02 18:12:30.472023 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.472000    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"]
Nov 02 18:12:30.477646 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.477616    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf75e7b-hk7n6\" (UID: \"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"
Nov 02 18:12:30.477739 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.477667    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf75e7b-hk7n6\" (UID: \"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"
Nov 02 18:12:30.477739 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.477722    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fg2pk\" (UniqueName: \"kubernetes.io/projected/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-kube-api-access-fg2pk\") pod \"maas-api-revoke-sub-5cf75e7b-hk7n6\" (UID: \"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"
Nov 02 18:12:30.484990 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.484379    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"]
Nov 02 18:12:30.487995 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.487968    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w2-f0a3b8-697b4d65c5-8w4g4"]
Nov 02 18:12:30.527538 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.527481    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"]
Nov 02 18:12:30.530656 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.530629    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w2-f0a3b8-67d6bfff99-r2cgz"]
Nov 02 18:12:30.546207 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.546178    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:30.578561 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.578533    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed882b3-d3b9-49c4-8bac-ef4aa5004535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fd065a0-2dxdl\" (UID: \"aed882b3-d3b9-49c4-8bac-ef4aa5004535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"
Nov 02 18:12:30.578714 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.578569    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf75e7b-hk7n6\" (UID: \"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"
Nov 02 18:12:30.578714 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.578606    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fg2pk\" (UniqueName: \"kubernetes.io/projected/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-kube-api-access-fg2pk\") pod \"maas-api-revoke-sub-5cf75e7b-hk7n6\" (UID: \"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"
Nov 02 18:12:30.578848 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.578737    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed882b3-d3b9-49c4-8bac-ef4aa5004535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fd065a0-2dxdl\" (UID: \"aed882b3-d3b9-49c4-8bac-ef4aa5004535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"
Nov 02 18:12:30.578910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.578862    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-27q6v\" (UniqueName: \"kubernetes.io/projected/aed882b3-d3b9-49c4-8bac-ef4aa5004535-kube-api-access-27q6v\") pod \"maas-api-revoke-sub-4fd065a0-2dxdl\" (UID: \"aed882b3-d3b9-49c4-8bac-ef4aa5004535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"
Nov 02 18:12:30.579014 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.579000    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf75e7b-hk7n6\" (UID: \"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"
Nov 02 18:12:30.579302 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.579280    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5cf75e7b-hk7n6\" (UID: \"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"
Nov 02 18:12:30.581511 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.581489    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5cf75e7b-hk7n6\" (UID: \"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"
Nov 02 18:12:30.603232 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.603205    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fg2pk\" (UniqueName: \"kubernetes.io/projected/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-kube-api-access-fg2pk\") pod \"maas-api-revoke-sub-5cf75e7b-hk7n6\" (UID: \"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"
Nov 02 18:12:30.658621 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.658537    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"]
Nov 02 18:12:30.661416 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.661398    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w1-120394-29qk8"]
Nov 02 18:12:30.679645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.679618    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed882b3-d3b9-49c4-8bac-ef4aa5004535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fd065a0-2dxdl\" (UID: \"aed882b3-d3b9-49c4-8bac-ef4aa5004535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"
Nov 02 18:12:30.679745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.679705    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed882b3-d3b9-49c4-8bac-ef4aa5004535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fd065a0-2dxdl\" (UID: \"aed882b3-d3b9-49c4-8bac-ef4aa5004535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"
Nov 02 18:12:30.679853 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.679758    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-27q6v\" (UniqueName: \"kubernetes.io/projected/aed882b3-d3b9-49c4-8bac-ef4aa5004535-kube-api-access-27q6v\") pod \"maas-api-revoke-sub-4fd065a0-2dxdl\" (UID: \"aed882b3-d3b9-49c4-8bac-ef4aa5004535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"
Nov 02 18:12:30.680310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.680289    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed882b3-d3b9-49c4-8bac-ef4aa5004535-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-4fd065a0-2dxdl\" (UID: \"aed882b3-d3b9-49c4-8bac-ef4aa5004535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"
Nov 02 18:12:30.682221 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.682200    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed882b3-d3b9-49c4-8bac-ef4aa5004535-maas-api-xks-ca\") pod \"maas-api-revoke-sub-4fd065a0-2dxdl\" (UID: \"aed882b3-d3b9-49c4-8bac-ef4aa5004535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"
Nov 02 18:12:30.693055 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.692788    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"
Nov 02 18:12:30.709034 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.708977    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-27q6v\" (UniqueName: \"kubernetes.io/projected/aed882b3-d3b9-49c4-8bac-ef4aa5004535-kube-api-access-27q6v\") pod \"maas-api-revoke-sub-4fd065a0-2dxdl\" (UID: \"aed882b3-d3b9-49c4-8bac-ef4aa5004535\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"
Nov 02 18:12:30.748400 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.748366    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:30.778583 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.778328    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"
Nov 02 18:12:30.926329 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.926179    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"]
Nov 02 18:12:30.928505 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:30.927502    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api" containerID="cri-o://39ff5c46888272351315f2d7d8139cbf2a129442737970d9469e7b5657c07974" gracePeriod=30
Nov 02 18:12:31.021899 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:12:31.021861    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfc4b930c_79d7_4beb_a5b0_0592c3bdcdd5.slice/crio-a7497e0bdd8eed0184d0b53f55a6d70fc3659a2a7a81e32d6906e102613fa273 WatchSource:0}: Error finding container a7497e0bdd8eed0184d0b53f55a6d70fc3659a2a7a81e32d6906e102613fa273: Status 404 returned error can't find the container with id a7497e0bdd8eed0184d0b53f55a6d70fc3659a2a7a81e32d6906e102613fa273
Nov 02 18:12:31.021899 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.021877    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"]
Nov 02 18:12:31.078153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.078121    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:31.102362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.100882    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"]
Nov 02 18:12:31.232170 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.232137    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:31.304133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.304109    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 18:12:31.311619 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.311588    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"]
Nov 02 18:12:31.314376 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.314355    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w4-f59572-98dfp"]
Nov 02 18:12:31.408348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.407442    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9051e10f-854f-4cc8-90a7-dda319c49c9a-metrics-certs\" (UniqueName: \"kubernetes.io/secret/9051e10f-854f-4cc8-90a7-dda319c49c9a-metrics-certs\") pod \"9051e10f-854f-4cc8-90a7-dda319c49c9a\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") "
Nov 02 18:12:31.408348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.407565    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9051e10f-854f-4cc8-90a7-dda319c49c9a-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9051e10f-854f-4cc8-90a7-dda319c49c9a-maas-api-tls\") pod \"9051e10f-854f-4cc8-90a7-dda319c49c9a\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") "
Nov 02 18:12:31.408348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.407597    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9051e10f-854f-4cc8-90a7-dda319c49c9a-kube-api-access-n4fff\" (UniqueName: \"kubernetes.io/projected/9051e10f-854f-4cc8-90a7-dda319c49c9a-kube-api-access-n4fff\") pod \"9051e10f-854f-4cc8-90a7-dda319c49c9a\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") "
Nov 02 18:12:31.408348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.407641    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9051e10f-854f-4cc8-90a7-dda319c49c9a-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9051e10f-854f-4cc8-90a7-dda319c49c9a-openshift-service-ca\") pod \"9051e10f-854f-4cc8-90a7-dda319c49c9a\" (UID: \"9051e10f-854f-4cc8-90a7-dda319c49c9a\") "
Nov 02 18:12:31.409162 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.408857    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9051e10f-854f-4cc8-90a7-dda319c49c9a-openshift-service-ca" pod "9051e10f-854f-4cc8-90a7-dda319c49c9a" (UID: "9051e10f-854f-4cc8-90a7-dda319c49c9a"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:31.430006 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.429962    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:31.439677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.438970    4456 generic.go:356] "Generic (PLEG): container finished" podID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerID="39ff5c46888272351315f2d7d8139cbf2a129442737970d9469e7b5657c07974" exitCode=0
Nov 02 18:12:31.439677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.439116    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" event={"ID":"9051e10f-854f-4cc8-90a7-dda319c49c9a","Type":"ContainerDied","Data":"39ff5c46888272351315f2d7d8139cbf2a129442737970d9469e7b5657c07974"}
Nov 02 18:12:31.439677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.439167    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m" event={"ID":"9051e10f-854f-4cc8-90a7-dda319c49c9a","Type":"ContainerDied","Data":"69a4f9bb8dd22a44673b4a64d84589a7ab3203631cde91ea4d3f286018af26f9"}
Nov 02 18:12:31.439677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.439198    4456 scope.go:122] "RemoveContainer" containerID="39ff5c46888272351315f2d7d8139cbf2a129442737970d9469e7b5657c07974"
Nov 02 18:12:31.439677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.439300    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"
Nov 02 18:12:31.442717 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.442692    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6" event={"ID":"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5","Type":"ContainerStarted","Data":"a7497e0bdd8eed0184d0b53f55a6d70fc3659a2a7a81e32d6906e102613fa273"}
Nov 02 18:12:31.444223 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.444187    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl" event={"ID":"aed882b3-d3b9-49c4-8bac-ef4aa5004535","Type":"ContainerStarted","Data":"19c658b35b86f506cc884a248450ec3af78ae5cf1c498a1ab91dcae5d875a613"}
Nov 02 18:12:31.448509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.448439    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9051e10f-854f-4cc8-90a7-dda319c49c9a-maas-api-tls" pod "9051e10f-854f-4cc8-90a7-dda319c49c9a" (UID: "9051e10f-854f-4cc8-90a7-dda319c49c9a"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:31.448637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.448486    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9051e10f-854f-4cc8-90a7-dda319c49c9a-kube-api-access-n4fff" pod "9051e10f-854f-4cc8-90a7-dda319c49c9a" (UID: "9051e10f-854f-4cc8-90a7-dda319c49c9a"). InnerVolumeSpecName "kube-api-access-n4fff". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:31.450340 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.450307    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9051e10f-854f-4cc8-90a7-dda319c49c9a-metrics-certs" pod "9051e10f-854f-4cc8-90a7-dda319c49c9a" (UID: "9051e10f-854f-4cc8-90a7-dda319c49c9a"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:31.462114 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.462053    4456 scope.go:122] "RemoveContainer" containerID="e4828365cff34649b08e97b25ad67554fb1835933f1008be8e43bed2f20e7560"
Nov 02 18:12:31.475783 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.475761    4456 scope.go:122] "RemoveContainer" containerID="39ff5c46888272351315f2d7d8139cbf2a129442737970d9469e7b5657c07974"
Nov 02 18:12:31.476396 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:31.476342    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"39ff5c46888272351315f2d7d8139cbf2a129442737970d9469e7b5657c07974\": container with ID starting with 39ff5c46888272351315f2d7d8139cbf2a129442737970d9469e7b5657c07974 not found: ID does not exist" containerID="39ff5c46888272351315f2d7d8139cbf2a129442737970d9469e7b5657c07974"
Nov 02 18:12:31.476491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.476421    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"39ff5c46888272351315f2d7d8139cbf2a129442737970d9469e7b5657c07974"} err="failed to get container status \"39ff5c46888272351315f2d7d8139cbf2a129442737970d9469e7b5657c07974\": rpc error: code = NotFound desc = could not find container \"39ff5c46888272351315f2d7d8139cbf2a129442737970d9469e7b5657c07974\": container with ID starting with 39ff5c46888272351315f2d7d8139cbf2a129442737970d9469e7b5657c07974 not found: ID does not exist"
Nov 02 18:12:31.476491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.476453    4456 scope.go:122] "RemoveContainer" containerID="e4828365cff34649b08e97b25ad67554fb1835933f1008be8e43bed2f20e7560"
Nov 02 18:12:31.476789 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:31.476759    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e4828365cff34649b08e97b25ad67554fb1835933f1008be8e43bed2f20e7560\": container with ID starting with e4828365cff34649b08e97b25ad67554fb1835933f1008be8e43bed2f20e7560 not found: ID does not exist" containerID="e4828365cff34649b08e97b25ad67554fb1835933f1008be8e43bed2f20e7560"
Nov 02 18:12:31.476968 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.476800    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e4828365cff34649b08e97b25ad67554fb1835933f1008be8e43bed2f20e7560"} err="failed to get container status \"e4828365cff34649b08e97b25ad67554fb1835933f1008be8e43bed2f20e7560\": rpc error: code = NotFound desc = could not find container \"e4828365cff34649b08e97b25ad67554fb1835933f1008be8e43bed2f20e7560\": container with ID starting with e4828365cff34649b08e97b25ad67554fb1835933f1008be8e43bed2f20e7560 not found: ID does not exist"
Nov 02 18:12:31.509395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.509371    4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/9051e10f-854f-4cc8-90a7-dda319c49c9a-metrics-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:31.509395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.509395    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/9051e10f-854f-4cc8-90a7-dda319c49c9a-maas-api-tls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:31.509516 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.509405    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-n4fff\" (UniqueName: \"kubernetes.io/projected/9051e10f-854f-4cc8-90a7-dda319c49c9a-kube-api-access-n4fff\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:31.509516 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.509414    4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/9051e10f-854f-4cc8-90a7-dda319c49c9a-openshift-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:31.630712 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.630665    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:31.633431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.633393    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18a32a7b-66fa-4f34-ab36-2beb0169e430" path="/var/lib/kubelet/pods/18a32a7b-66fa-4f34-ab36-2beb0169e430/volumes"
Nov 02 18:12:31.634052 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.634023    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2c58a7cf-d0de-4692-bc36-7d02d346eb2e" path="/var/lib/kubelet/pods/2c58a7cf-d0de-4692-bc36-7d02d346eb2e/volumes"
Nov 02 18:12:31.634648 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.634628    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a230f129-4b8a-4544-9e35-053804d1edd7" path="/var/lib/kubelet/pods/a230f129-4b8a-4544-9e35-053804d1edd7/volumes"
Nov 02 18:12:31.635591 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.635570    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b5d45c38-1cef-4aa9-a519-98aed00e8727" path="/var/lib/kubelet/pods/b5d45c38-1cef-4aa9-a519-98aed00e8727/volumes"
Nov 02 18:12:31.636155 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.636135    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d99092cc-b126-48db-8762-a4b8d64ad350" path="/var/lib/kubelet/pods/d99092cc-b126-48db-8762-a4b8d64ad350/volumes"
Nov 02 18:12:31.636680 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.636662    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e869a01e-b2dc-414c-949b-f86d71c428be" path="/var/lib/kubelet/pods/e869a01e-b2dc-414c-949b-f86d71c428be/volumes"
Nov 02 18:12:31.787612 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.787474    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:31.788300 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.788274    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"]
Nov 02 18:12:31.792248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.792221    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-a-w5-504de7-d95d58fbd-h8d2m"]
Nov 02 18:12:31.826745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.826710    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"]
Nov 02 18:12:31.845357 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.845268    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w5-504de7-mrvkv"]
Nov 02 18:12:31.950770 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:31.950732    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:32.194163 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.194137    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:32.297477 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.297434    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"]
Nov 02 18:12:32.300273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.300244    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w3-557feb-msqtv"]
Nov 02 18:12:32.362193 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.362161    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:32.456250 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.456218    4456 generic.go:356] "Generic (PLEG): container finished" podID="fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5" containerID="44af70e06419b72cee419c8f1a1c535d95e4012f225cd305a78ac8ee937de357" exitCode=0
Nov 02 18:12:32.456429 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.456288    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6" event={"ID":"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5","Type":"ContainerDied","Data":"44af70e06419b72cee419c8f1a1c535d95e4012f225cd305a78ac8ee937de357"}
Nov 02 18:12:32.457959 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.457934    4456 generic.go:356] "Generic (PLEG): container finished" podID="aed882b3-d3b9-49c4-8bac-ef4aa5004535" containerID="08f8867e9ea760910a33ddc37130b5a36993599b9b7b82fa40374f018983d586" exitCode=0
Nov 02 18:12:32.458049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.457971    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl" event={"ID":"aed882b3-d3b9-49c4-8bac-ef4aa5004535","Type":"ContainerDied","Data":"08f8867e9ea760910a33ddc37130b5a36993599b9b7b82fa40374f018983d586"}
Nov 02 18:12:32.506946 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.506913    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:32.772146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.772081    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"]
Nov 02 18:12:32.772562 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.772551    4456 cpu_manager.go:414] "containerMap: removing container" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api"
Nov 02 18:12:32.772603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.772564    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api"
Nov 02 18:12:32.772603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.772583    4456 cpu_manager.go:414] "containerMap: removing container" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api"
Nov 02 18:12:32.772603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.772588    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api"
Nov 02 18:12:32.772688 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.772616    4456 cpu_manager.go:414] "containerMap: removing container" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api"
Nov 02 18:12:32.772688 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.772621    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api"
Nov 02 18:12:32.772745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.772704    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api"
Nov 02 18:12:32.772745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.772720    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api"
Nov 02 18:12:32.776010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.775995    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.796098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.796073    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c4dbe3\"/\"kube-root-ca.crt\""
Nov 02 18:12:32.796515 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.796236    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"ai-tenant-e2e-shared-b-w0-c4dbe3\"/\"openshift-service-ca.crt\""
Nov 02 18:12:32.796515 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.796252    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c4dbe3\"/\"auth-test-model-c4dbe3-kserve-self-signed-certs\""
Nov 02 18:12:32.796515 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.796282    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"ai-tenant-e2e-shared-b-w0-c4dbe3\"/\"default-dockercfg-7xsbm\""
Nov 02 18:12:32.799089 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.799065    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"]
Nov 02 18:12:32.824912 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.824880    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-tmp-dir\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.825111 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.825092    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zzqkn\" (UniqueName: \"kubernetes.io/projected/3a915d83-872d-4022-9e74-f2fa93aeb737-kube-api-access-zzqkn\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.825180 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.825152    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-model-cache\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.825289 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.825269    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-dshm\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.825350 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.825310    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-home\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.825441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.825417    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a915d83-872d-4022-9e74-f2fa93aeb737-tls-certs\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.842508 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.842486    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:32.927102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.927061    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zzqkn\" (UniqueName: \"kubernetes.io/projected/3a915d83-872d-4022-9e74-f2fa93aeb737-kube-api-access-zzqkn\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.927287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.927141    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-model-cache\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.927287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.927176    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-dshm\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.927287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.927206    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-home\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.927287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.927245    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a915d83-872d-4022-9e74-f2fa93aeb737-tls-certs\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.927484 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.927324    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-tmp-dir\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.927646 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.927619    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-model-cache\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.927708 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.927690    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-home\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.927765 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.927709    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-tmp-dir\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.929807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.929775    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-dshm\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.930172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.930153    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a915d83-872d-4022-9e74-f2fa93aeb737-tls-certs\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:32.952796 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:32.952776    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zzqkn\" (UniqueName: \"kubernetes.io/projected/3a915d83-872d-4022-9e74-f2fa93aeb737-kube-api-access-zzqkn\") pod \"auth-test-model-c4dbe3-kserve-65c444678f-vwv24\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") " pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:33.008925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.008896    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:33.090923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.089370    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:33.402428 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.402399    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"]
Nov 02 18:12:33.402666 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:12:33.402642    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod3a915d83_872d_4022_9e74_f2fa93aeb737.slice/crio-ac874e35b60759f8b57acad698f1b105105d0891f501fe769320f05ff826eea4 WatchSource:0}: Error finding container ac874e35b60759f8b57acad698f1b105105d0891f501fe769320f05ff826eea4: Status 404 returned error can't find the container with id ac874e35b60759f8b57acad698f1b105105d0891f501fe769320f05ff826eea4
Nov 02 18:12:33.466514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.466476    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24" event={"ID":"3a915d83-872d-4022-9e74-f2fa93aeb737","Type":"ContainerStarted","Data":"ac874e35b60759f8b57acad698f1b105105d0891f501fe769320f05ff826eea4"}
Nov 02 18:12:33.628841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.628528    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"
Nov 02 18:12:33.629564 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.629536    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8045e931-2116-422f-806e-f18e3b757a0c" path="/var/lib/kubelet/pods/8045e931-2116-422f-806e-f18e3b757a0c/volumes"
Nov 02 18:12:33.630071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.630051    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" path="/var/lib/kubelet/pods/9051e10f-854f-4cc8-90a7-dda319c49c9a/volumes"
Nov 02 18:12:33.630553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.630537    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e2442a9b-67dd-4a54-bd70-c9075e1a6522" path="/var/lib/kubelet/pods/e2442a9b-67dd-4a54-bd70-c9075e1a6522/volumes"
Nov 02 18:12:33.649711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.649659    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"
Nov 02 18:12:33.736973 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.736944    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-maas-api-openshift-ca\") pod \"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5\" (UID: \"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5\") "
Nov 02 18:12:33.737143 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.736995    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/aed882b3-d3b9-49c4-8bac-ef4aa5004535-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed882b3-d3b9-49c4-8bac-ef4aa5004535-maas-api-xks-ca\") pod \"aed882b3-d3b9-49c4-8bac-ef4aa5004535\" (UID: \"aed882b3-d3b9-49c4-8bac-ef4aa5004535\") "
Nov 02 18:12:33.737220 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.737144    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-maas-api-xks-ca\") pod \"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5\" (UID: \"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5\") "
Nov 02 18:12:33.737220 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.737175    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-kube-api-access-fg2pk\" (UniqueName: \"kubernetes.io/projected/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-kube-api-access-fg2pk\") pod \"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5\" (UID: \"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5\") "
Nov 02 18:12:33.737320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.737255    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/aed882b3-d3b9-49c4-8bac-ef4aa5004535-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed882b3-d3b9-49c4-8bac-ef4aa5004535-maas-api-openshift-ca\") pod \"aed882b3-d3b9-49c4-8bac-ef4aa5004535\" (UID: \"aed882b3-d3b9-49c4-8bac-ef4aa5004535\") "
Nov 02 18:12:33.737320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.737299    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/aed882b3-d3b9-49c4-8bac-ef4aa5004535-kube-api-access-27q6v\" (UniqueName: \"kubernetes.io/projected/aed882b3-d3b9-49c4-8bac-ef4aa5004535-kube-api-access-27q6v\") pod \"aed882b3-d3b9-49c4-8bac-ef4aa5004535\" (UID: \"aed882b3-d3b9-49c4-8bac-ef4aa5004535\") "
Nov 02 18:12:33.737421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.737330    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-maas-api-openshift-ca" pod "fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5" (UID: "fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:33.737676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.737653    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/aed882b3-d3b9-49c4-8bac-ef4aa5004535-maas-api-openshift-ca" pod "aed882b3-d3b9-49c4-8bac-ef4aa5004535" (UID: "aed882b3-d3b9-49c4-8bac-ef4aa5004535"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:33.738101 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.738081    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/aed882b3-d3b9-49c4-8bac-ef4aa5004535-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:33.738201 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.738131    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:33.740551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.740525    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/aed882b3-d3b9-49c4-8bac-ef4aa5004535-kube-api-access-27q6v" pod "aed882b3-d3b9-49c4-8bac-ef4aa5004535" (UID: "aed882b3-d3b9-49c4-8bac-ef4aa5004535"). InnerVolumeSpecName "kube-api-access-27q6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:33.740680 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.740556    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/aed882b3-d3b9-49c4-8bac-ef4aa5004535-maas-api-xks-ca" pod "aed882b3-d3b9-49c4-8bac-ef4aa5004535" (UID: "aed882b3-d3b9-49c4-8bac-ef4aa5004535"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:33.741095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.741009    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-kube-api-access-fg2pk" pod "fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5" (UID: "fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5"). InnerVolumeSpecName "kube-api-access-fg2pk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:33.741095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.741061    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-maas-api-xks-ca" pod "fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5" (UID: "fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:33.839619 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.839585    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:33.839619 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.839614    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fg2pk\" (UniqueName: \"kubernetes.io/projected/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5-kube-api-access-fg2pk\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:33.839619 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.839626    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-27q6v\" (UniqueName: \"kubernetes.io/projected/aed882b3-d3b9-49c4-8bac-ef4aa5004535-kube-api-access-27q6v\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:33.840079 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:33.839636    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/aed882b3-d3b9-49c4-8bac-ef4aa5004535-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:34.139000 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:34.138960    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:34.475561 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:34.475534    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"
Nov 02 18:12:34.475756 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:34.475535    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6" event={"ID":"fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5","Type":"ContainerDied","Data":"a7497e0bdd8eed0184d0b53f55a6d70fc3659a2a7a81e32d6906e102613fa273"}
Nov 02 18:12:34.475756 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:34.475668    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="a7497e0bdd8eed0184d0b53f55a6d70fc3659a2a7a81e32d6906e102613fa273"
Nov 02 18:12:34.477018 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:34.477000    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"
Nov 02 18:12:34.477115 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:34.477035    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl" event={"ID":"aed882b3-d3b9-49c4-8bac-ef4aa5004535","Type":"ContainerDied","Data":"19c658b35b86f506cc884a248450ec3af78ae5cf1c498a1ab91dcae5d875a613"}
Nov 02 18:12:34.477115 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:34.477076    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="19c658b35b86f506cc884a248450ec3af78ae5cf1c498a1ab91dcae5d875a613"
Nov 02 18:12:34.478969 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:34.478946    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24" event={"ID":"3a915d83-872d-4022-9e74-f2fa93aeb737","Type":"ContainerStarted","Data":"3878b633a8eaa74e610fcee33456a6cf5b3c7971162acca80149d55649d705c0"}
Nov 02 18:12:34.534489 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:34.534438    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24" podStartSLOduration=2.5344246950000002 podStartE2EDuration="2.534424695s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:12:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:12:34.531683555 +0000 UTC m=+2571.022746716" watchObservedRunningTime="2026-11-02 18:12:34.534424695 +0000 UTC m=+2571.025487856"
Nov 02 18:12:34.603711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:34.603676    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:34.761338 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:34.761252    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:34.900370 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:34.900338    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:35.090599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.090520    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:35.100585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.100556    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:35.634721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.634690    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"]
Nov 02 18:12:35.635206 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.635192    4456 cpu_manager.go:414] "containerMap: removing container" podUID="aed882b3-d3b9-49c4-8bac-ef4aa5004535" containerName="revoke-keys"
Nov 02 18:12:35.635245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.635208    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="aed882b3-d3b9-49c4-8bac-ef4aa5004535" containerName="revoke-keys"
Nov 02 18:12:35.635245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.635217    4456 cpu_manager.go:414] "containerMap: removing container" podUID="fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5" containerName="revoke-keys"
Nov 02 18:12:35.635245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.635222    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5" containerName="revoke-keys"
Nov 02 18:12:35.635330 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.635291    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="aed882b3-d3b9-49c4-8bac-ef4aa5004535" containerName="revoke-keys"
Nov 02 18:12:35.635330 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.635299    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5" containerName="revoke-keys"
Nov 02 18:12:35.635330 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.635309    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="9051e10f-854f-4cc8-90a7-dda319c49c9a" containerName="maas-api"
Nov 02 18:12:35.639699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.639684    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"
Nov 02 18:12:35.659093 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.659061    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"]
Nov 02 18:12:35.667622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.667587    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-pvwwj\" (UniqueName: \"kubernetes.io/projected/72eb6022-bf9b-43da-933a-b6fae2f5c142-kube-api-access-pvwwj\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w\" (UID: \"72eb6022-bf9b-43da-933a-b6fae2f5c142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"
Nov 02 18:12:35.667831 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.667699    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72eb6022-bf9b-43da-933a-b6fae2f5c142-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w\" (UID: \"72eb6022-bf9b-43da-933a-b6fae2f5c142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"
Nov 02 18:12:35.668007 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.667976    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72eb6022-bf9b-43da-933a-b6fae2f5c142-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w\" (UID: \"72eb6022-bf9b-43da-933a-b6fae2f5c142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"
Nov 02 18:12:35.668174 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.668150    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:12:35.668284 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.668269    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:12:35.769340 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.769308    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72eb6022-bf9b-43da-933a-b6fae2f5c142-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w\" (UID: \"72eb6022-bf9b-43da-933a-b6fae2f5c142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"
Nov 02 18:12:35.769526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.769418    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-pvwwj\" (UniqueName: \"kubernetes.io/projected/72eb6022-bf9b-43da-933a-b6fae2f5c142-kube-api-access-pvwwj\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w\" (UID: \"72eb6022-bf9b-43da-933a-b6fae2f5c142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"
Nov 02 18:12:35.769526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.769477    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72eb6022-bf9b-43da-933a-b6fae2f5c142-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w\" (UID: \"72eb6022-bf9b-43da-933a-b6fae2f5c142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"
Nov 02 18:12:35.770203 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.770173    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72eb6022-bf9b-43da-933a-b6fae2f5c142-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w\" (UID: \"72eb6022-bf9b-43da-933a-b6fae2f5c142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"
Nov 02 18:12:35.772365 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.772340    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72eb6022-bf9b-43da-933a-b6fae2f5c142-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w\" (UID: \"72eb6022-bf9b-43da-933a-b6fae2f5c142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"
Nov 02 18:12:35.795386 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.795359    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-pvwwj\" (UniqueName: \"kubernetes.io/projected/72eb6022-bf9b-43da-933a-b6fae2f5c142-kube-api-access-pvwwj\") pod \"maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w\" (UID: \"72eb6022-bf9b-43da-933a-b6fae2f5c142\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"
Nov 02 18:12:35.952914 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:35.952885    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"
Nov 02 18:12:36.122389 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:36.122313    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:36.258303 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:36.258266    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"]
Nov 02 18:12:36.258727 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:12:36.258696    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod72eb6022_bf9b_43da_933a_b6fae2f5c142.slice/crio-39b0359705794be7a94aad9711e144bbb4a1c2e8321c0c599fbd2aefc0e0aad6 WatchSource:0}: Error finding container 39b0359705794be7a94aad9711e144bbb4a1c2e8321c0c599fbd2aefc0e0aad6: Status 404 returned error can't find the container with id 39b0359705794be7a94aad9711e144bbb4a1c2e8321c0c599fbd2aefc0e0aad6
Nov 02 18:12:36.495261 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:36.495227    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w" event={"ID":"72eb6022-bf9b-43da-933a-b6fae2f5c142","Type":"ContainerStarted","Data":"39b0359705794be7a94aad9711e144bbb4a1c2e8321c0c599fbd2aefc0e0aad6"}
Nov 02 18:12:37.129750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:37.129603    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:37.461420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:37.461379    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:37.506114 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:37.506078    4456 generic.go:356] "Generic (PLEG): container finished" podID="72eb6022-bf9b-43da-933a-b6fae2f5c142" containerID="401ebf16b44e5436d527807df450685a251b2fed001cdf07320abf6b9bf2eb1b" exitCode=0
Nov 02 18:12:37.506280 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:37.506160    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w" event={"ID":"72eb6022-bf9b-43da-933a-b6fae2f5c142","Type":"ContainerDied","Data":"401ebf16b44e5436d527807df450685a251b2fed001cdf07320abf6b9bf2eb1b"}
Nov 02 18:12:37.700276 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:37.700243    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:37.903607 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:37.903525    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:38.120155 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:38.120103    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:38.368780 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:38.368745    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:38.602337 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:38.601328    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:38.691316 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:38.691289    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"
Nov 02 18:12:38.808725 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:38.808684    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/72eb6022-bf9b-43da-933a-b6fae2f5c142-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72eb6022-bf9b-43da-933a-b6fae2f5c142-maas-api-xks-ca\") pod \"72eb6022-bf9b-43da-933a-b6fae2f5c142\" (UID: \"72eb6022-bf9b-43da-933a-b6fae2f5c142\") "
Nov 02 18:12:38.808942 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:38.808791    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/72eb6022-bf9b-43da-933a-b6fae2f5c142-kube-api-access-pvwwj\" (UniqueName: \"kubernetes.io/projected/72eb6022-bf9b-43da-933a-b6fae2f5c142-kube-api-access-pvwwj\") pod \"72eb6022-bf9b-43da-933a-b6fae2f5c142\" (UID: \"72eb6022-bf9b-43da-933a-b6fae2f5c142\") "
Nov 02 18:12:38.808942 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:38.808870    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/72eb6022-bf9b-43da-933a-b6fae2f5c142-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72eb6022-bf9b-43da-933a-b6fae2f5c142-maas-api-openshift-ca\") pod \"72eb6022-bf9b-43da-933a-b6fae2f5c142\" (UID: \"72eb6022-bf9b-43da-933a-b6fae2f5c142\") "
Nov 02 18:12:38.809291 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:38.809261    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/72eb6022-bf9b-43da-933a-b6fae2f5c142-maas-api-openshift-ca" pod "72eb6022-bf9b-43da-933a-b6fae2f5c142" (UID: "72eb6022-bf9b-43da-933a-b6fae2f5c142"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:38.809408 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:38.809336    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/72eb6022-bf9b-43da-933a-b6fae2f5c142-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:38.811233 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:38.811204    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/72eb6022-bf9b-43da-933a-b6fae2f5c142-kube-api-access-pvwwj" pod "72eb6022-bf9b-43da-933a-b6fae2f5c142" (UID: "72eb6022-bf9b-43da-933a-b6fae2f5c142"). InnerVolumeSpecName "kube-api-access-pvwwj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:38.811497 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:38.811475    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/72eb6022-bf9b-43da-933a-b6fae2f5c142-maas-api-xks-ca" pod "72eb6022-bf9b-43da-933a-b6fae2f5c142" (UID: "72eb6022-bf9b-43da-933a-b6fae2f5c142"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:38.910214 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:38.910134    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/72eb6022-bf9b-43da-933a-b6fae2f5c142-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:38.910214 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:38.910162    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-pvwwj\" (UniqueName: \"kubernetes.io/projected/72eb6022-bf9b-43da-933a-b6fae2f5c142-kube-api-access-pvwwj\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:39.184802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:39.184765    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:39.524680 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:39.524591    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"
Nov 02 18:12:39.524680 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:39.524617    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w" event={"ID":"72eb6022-bf9b-43da-933a-b6fae2f5c142","Type":"ContainerDied","Data":"39b0359705794be7a94aad9711e144bbb4a1c2e8321c0c599fbd2aefc0e0aad6"}
Nov 02 18:12:39.525137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:39.524708    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="39b0359705794be7a94aad9711e144bbb4a1c2e8321c0c599fbd2aefc0e0aad6"
Nov 02 18:12:39.902213 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:39.902125    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:40.586013 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:40.585984    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:40.740986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:40.740949    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:40.907884 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:40.907784    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:41.057005 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.056957    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:41.199986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.199948    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:41.446419 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.446387    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"]
Nov 02 18:12:41.447917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.447886    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy" containerID="cri-o://5d1811fa76419aa50beae48550ad5ab8de1c9137d28d84239f25338279ddf29d" gracePeriod=30
Nov 02 18:12:41.507672 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.507584    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"]
Nov 02 18:12:41.508121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.508101    4456 cpu_manager.go:414] "containerMap: removing container" podUID="72eb6022-bf9b-43da-933a-b6fae2f5c142" containerName="revoke-keys"
Nov 02 18:12:41.508121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.508116    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="72eb6022-bf9b-43da-933a-b6fae2f5c142" containerName="revoke-keys"
Nov 02 18:12:41.508310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.508205    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="72eb6022-bf9b-43da-933a-b6fae2f5c142" containerName="revoke-keys"
Nov 02 18:12:41.511797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.511774    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"
Nov 02 18:12:41.530192 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.530166    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:12:41.530890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.530305    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:12:41.534038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.534016    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"]
Nov 02 18:12:41.534937 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.534911    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:41.643795 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.643725    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l57hr\" (UniqueName: \"kubernetes.io/projected/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-kube-api-access-l57hr\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf\" (UID: \"2e0ba0a6-de17-49db-91c5-4d4fe845aa67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"
Nov 02 18:12:41.644378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.643960    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf\" (UID: \"2e0ba0a6-de17-49db-91c5-4d4fe845aa67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"
Nov 02 18:12:41.644378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.644007    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf\" (UID: \"2e0ba0a6-de17-49db-91c5-4d4fe845aa67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"
Nov 02 18:12:41.745725 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.745693    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l57hr\" (UniqueName: \"kubernetes.io/projected/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-kube-api-access-l57hr\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf\" (UID: \"2e0ba0a6-de17-49db-91c5-4d4fe845aa67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"
Nov 02 18:12:41.745945 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.745810    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf\" (UID: \"2e0ba0a6-de17-49db-91c5-4d4fe845aa67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"
Nov 02 18:12:41.746022 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.745972    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf\" (UID: \"2e0ba0a6-de17-49db-91c5-4d4fe845aa67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"
Nov 02 18:12:41.746489 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.746469    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf\" (UID: \"2e0ba0a6-de17-49db-91c5-4d4fe845aa67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"
Nov 02 18:12:41.748527 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.748505    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf\" (UID: \"2e0ba0a6-de17-49db-91c5-4d4fe845aa67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"
Nov 02 18:12:41.752161 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.752133    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:41.771369 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.771308    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l57hr\" (UniqueName: \"kubernetes.io/projected/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-kube-api-access-l57hr\") pod \"maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf\" (UID: \"2e0ba0a6-de17-49db-91c5-4d4fe845aa67\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"
Nov 02 18:12:41.825753 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.825727    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"
Nov 02 18:12:41.917841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:41.915672    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:42.153081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.153029    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"]
Nov 02 18:12:42.206265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.206237    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:42.481876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.481835    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:42.550305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.550268    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf" event={"ID":"2e0ba0a6-de17-49db-91c5-4d4fe845aa67","Type":"ContainerStarted","Data":"9908738740a4d442ba29a6ddac0a89a05fc249ad9c39bf216ab06f50cf444344"}
Nov 02 18:12:42.550453 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.550366    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf" event={"ID":"2e0ba0a6-de17-49db-91c5-4d4fe845aa67","Type":"ContainerStarted","Data":"ee3811fc238fc1e3ce65b55dfe75a3b6943054e8c7d2119da7b4d51851e62540"}
Nov 02 18:12:42.609171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.609137    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"]
Nov 02 18:12:42.609420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.609394    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy" containerID="cri-o://43b444ecc72b8f3eef31fafa69b45fd05eaa0fc026c197971824e9e9553d8fe9" gracePeriod=30
Nov 02 18:12:42.610328 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.610282    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf" podStartSLOduration=1.610270372 podStartE2EDuration="1.610270372s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:12:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:12:42.606373539 +0000 UTC m=+2579.097436701" watchObservedRunningTime="2026-11-02 18:12:42.610270372 +0000 UTC m=+2579.101333560"
Nov 02 18:12:42.688604 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.688573    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:42.737592 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.737511    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"]
Nov 02 18:12:42.742222 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.742197    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"
Nov 02 18:12:42.762991 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.762963    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"]
Nov 02 18:12:42.872553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.872521    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714ec0a1-9652-4524-81d9-1ca9dc846a56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m\" (UID: \"714ec0a1-9652-4524-81d9-1ca9dc846a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"
Nov 02 18:12:42.872696 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.872563    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714ec0a1-9652-4524-81d9-1ca9dc846a56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m\" (UID: \"714ec0a1-9652-4524-81d9-1ca9dc846a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"
Nov 02 18:12:42.872737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.872725    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-9bzq6\" (UniqueName: \"kubernetes.io/projected/714ec0a1-9652-4524-81d9-1ca9dc846a56-kube-api-access-9bzq6\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m\" (UID: \"714ec0a1-9652-4524-81d9-1ca9dc846a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"
Nov 02 18:12:42.974095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.974058    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-9bzq6\" (UniqueName: \"kubernetes.io/projected/714ec0a1-9652-4524-81d9-1ca9dc846a56-kube-api-access-9bzq6\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m\" (UID: \"714ec0a1-9652-4524-81d9-1ca9dc846a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"
Nov 02 18:12:42.974242 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.974154    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714ec0a1-9652-4524-81d9-1ca9dc846a56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m\" (UID: \"714ec0a1-9652-4524-81d9-1ca9dc846a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"
Nov 02 18:12:42.974242 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.974181    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714ec0a1-9652-4524-81d9-1ca9dc846a56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m\" (UID: \"714ec0a1-9652-4524-81d9-1ca9dc846a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"
Nov 02 18:12:42.974796 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.974773    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714ec0a1-9652-4524-81d9-1ca9dc846a56-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m\" (UID: \"714ec0a1-9652-4524-81d9-1ca9dc846a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"
Nov 02 18:12:42.976715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.976692    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714ec0a1-9652-4524-81d9-1ca9dc846a56-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m\" (UID: \"714ec0a1-9652-4524-81d9-1ca9dc846a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"
Nov 02 18:12:42.999298 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:42.999237    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-9bzq6\" (UniqueName: \"kubernetes.io/projected/714ec0a1-9652-4524-81d9-1ca9dc846a56-kube-api-access-9bzq6\") pod \"maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m\" (UID: \"714ec0a1-9652-4524-81d9-1ca9dc846a56\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"
Nov 02 18:12:43.072909 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.072877    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"
Nov 02 18:12:43.090021 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.089993    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:43.104211 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.104190    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:43.152665 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.152633    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"]
Nov 02 18:12:43.156993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.156972    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"
Nov 02 18:12:43.180373 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.180342    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"]
Nov 02 18:12:43.277416 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.277335    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg\" (UID: \"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"
Nov 02 18:12:43.277555 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.277534    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg\" (UID: \"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"
Nov 02 18:12:43.277663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.277645    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fdp88\" (UniqueName: \"kubernetes.io/projected/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-kube-api-access-fdp88\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg\" (UID: \"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"
Nov 02 18:12:43.375251 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:12:43.375222    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod714ec0a1_9652_4524_81d9_1ca9dc846a56.slice/crio-c153ce64ebce801d160b4539afd375a7c2872c04fb7b7b2620f918544d280bb8 WatchSource:0}: Error finding container c153ce64ebce801d160b4539afd375a7c2872c04fb7b7b2620f918544d280bb8: Status 404 returned error can't find the container with id c153ce64ebce801d160b4539afd375a7c2872c04fb7b7b2620f918544d280bb8
Nov 02 18:12:43.375554 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.375529    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"]
Nov 02 18:12:43.378368 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.378348    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg\" (UID: \"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"
Nov 02 18:12:43.378460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.378442    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fdp88\" (UniqueName: \"kubernetes.io/projected/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-kube-api-access-fdp88\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg\" (UID: \"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"
Nov 02 18:12:43.378500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.378484    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg\" (UID: \"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"
Nov 02 18:12:43.378984 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.378966    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg\" (UID: \"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"
Nov 02 18:12:43.381174 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.381154    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg\" (UID: \"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"
Nov 02 18:12:43.403147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.403127    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fdp88\" (UniqueName: \"kubernetes.io/projected/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-kube-api-access-fdp88\") pod \"maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg\" (UID: \"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"
Nov 02 18:12:43.471710 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.471676    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"
Nov 02 18:12:43.560287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.560254    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m" event={"ID":"714ec0a1-9652-4524-81d9-1ca9dc846a56","Type":"ContainerStarted","Data":"c153ce64ebce801d160b4539afd375a7c2872c04fb7b7b2620f918544d280bb8"}
Nov 02 18:12:43.562116 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.562089    4456 generic.go:356] "Generic (PLEG): container finished" podID="2e0ba0a6-de17-49db-91c5-4d4fe845aa67" containerID="9908738740a4d442ba29a6ddac0a89a05fc249ad9c39bf216ab06f50cf444344" exitCode=0
Nov 02 18:12:43.562247 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.562175    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf" event={"ID":"2e0ba0a6-de17-49db-91c5-4d4fe845aa67","Type":"ContainerDied","Data":"9908738740a4d442ba29a6ddac0a89a05fc249ad9c39bf216ab06f50cf444344"}
Nov 02 18:12:43.578896 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.578872    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:12:43.774883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.774852    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"]
Nov 02 18:12:43.775277 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:12:43.775151    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f3984cc_87b7_4b39_9a93_d0f0961fc5d4.slice/crio-b8807c56ebd83df739bde06828872d77101c778cdfff98f737e46da046826d36 WatchSource:0}: Error finding container b8807c56ebd83df739bde06828872d77101c778cdfff98f737e46da046826d36: Status 404 returned error can't find the container with id b8807c56ebd83df739bde06828872d77101c778cdfff98f737e46da046826d36
Nov 02 18:12:43.980124 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:43.980093    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:44.570726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:44.570688    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg" event={"ID":"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4","Type":"ContainerStarted","Data":"45c8454f3d685f36d4e8376892c0d02950b001de66596fa7dd338b90f71d8aef"}
Nov 02 18:12:44.570929 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:44.570751    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg" event={"ID":"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4","Type":"ContainerStarted","Data":"b8807c56ebd83df739bde06828872d77101c778cdfff98f737e46da046826d36"}
Nov 02 18:12:44.572608 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:44.572578    4456 generic.go:356] "Generic (PLEG): container finished" podID="714ec0a1-9652-4524-81d9-1ca9dc846a56" containerID="e8183aab1c9e706d4a6b3dea7ef363ecbf7ba4bdb05426c127e5ed0525bee425" exitCode=0
Nov 02 18:12:44.572737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:44.572636    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m" event={"ID":"714ec0a1-9652-4524-81d9-1ca9dc846a56","Type":"ContainerDied","Data":"e8183aab1c9e706d4a6b3dea7ef363ecbf7ba4bdb05426c127e5ed0525bee425"}
Nov 02 18:12:44.627419 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:44.627367    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg" podStartSLOduration=1.6273518930000002 podStartE2EDuration="1.627351893s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:12:43 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:12:44.613198715 +0000 UTC m=+2581.104261895" watchObservedRunningTime="2026-11-02 18:12:44.627351893 +0000 UTC m=+2581.118415054"
Nov 02 18:12:44.722471 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:44.722445    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"
Nov 02 18:12:44.796983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:44.796944    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-kube-api-access-l57hr\" (UniqueName: \"kubernetes.io/projected/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-kube-api-access-l57hr\") pod \"2e0ba0a6-de17-49db-91c5-4d4fe845aa67\" (UID: \"2e0ba0a6-de17-49db-91c5-4d4fe845aa67\") "
Nov 02 18:12:44.797456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:44.797127    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-maas-api-openshift-ca\") pod \"2e0ba0a6-de17-49db-91c5-4d4fe845aa67\" (UID: \"2e0ba0a6-de17-49db-91c5-4d4fe845aa67\") "
Nov 02 18:12:44.797456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:44.797172    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-maas-api-xks-ca\") pod \"2e0ba0a6-de17-49db-91c5-4d4fe845aa67\" (UID: \"2e0ba0a6-de17-49db-91c5-4d4fe845aa67\") "
Nov 02 18:12:44.797718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:44.797629    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-maas-api-openshift-ca" pod "2e0ba0a6-de17-49db-91c5-4d4fe845aa67" (UID: "2e0ba0a6-de17-49db-91c5-4d4fe845aa67"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:44.799339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:44.799310    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-kube-api-access-l57hr" pod "2e0ba0a6-de17-49db-91c5-4d4fe845aa67" (UID: "2e0ba0a6-de17-49db-91c5-4d4fe845aa67"). InnerVolumeSpecName "kube-api-access-l57hr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:44.799470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:44.799446    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-maas-api-xks-ca" pod "2e0ba0a6-de17-49db-91c5-4d4fe845aa67" (UID: "2e0ba0a6-de17-49db-91c5-4d4fe845aa67"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:44.898730 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:44.898693    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:44.898730 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:44.898720    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:44.898730 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:44.898737    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l57hr\" (UniqueName: \"kubernetes.io/projected/2e0ba0a6-de17-49db-91c5-4d4fe845aa67-kube-api-access-l57hr\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:45.542919 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.542887    4456 patch_prober.go:29] interesting pod/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:12:45.543067 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.542944    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:12:45.581722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.581688    4456 generic.go:356] "Generic (PLEG): container finished" podID="7f3984cc-87b7-4b39-9a93-d0f0961fc5d4" containerID="45c8454f3d685f36d4e8376892c0d02950b001de66596fa7dd338b90f71d8aef" exitCode=0
Nov 02 18:12:45.581923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.581776    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg" event={"ID":"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4","Type":"ContainerDied","Data":"45c8454f3d685f36d4e8376892c0d02950b001de66596fa7dd338b90f71d8aef"}
Nov 02 18:12:45.583449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.583416    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf" event={"ID":"2e0ba0a6-de17-49db-91c5-4d4fe845aa67","Type":"ContainerDied","Data":"ee3811fc238fc1e3ce65b55dfe75a3b6943054e8c7d2119da7b4d51851e62540"}
Nov 02 18:12:45.583567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.583466    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ee3811fc238fc1e3ce65b55dfe75a3b6943054e8c7d2119da7b4d51851e62540"
Nov 02 18:12:45.583567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.583560    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"
Nov 02 18:12:45.751751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.751722    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"
Nov 02 18:12:45.790677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.790615    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:45.910150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.910062    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/714ec0a1-9652-4524-81d9-1ca9dc846a56-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714ec0a1-9652-4524-81d9-1ca9dc846a56-maas-api-openshift-ca\") pod \"714ec0a1-9652-4524-81d9-1ca9dc846a56\" (UID: \"714ec0a1-9652-4524-81d9-1ca9dc846a56\") "
Nov 02 18:12:45.910620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.910257    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/714ec0a1-9652-4524-81d9-1ca9dc846a56-kube-api-access-9bzq6\" (UniqueName: \"kubernetes.io/projected/714ec0a1-9652-4524-81d9-1ca9dc846a56-kube-api-access-9bzq6\") pod \"714ec0a1-9652-4524-81d9-1ca9dc846a56\" (UID: \"714ec0a1-9652-4524-81d9-1ca9dc846a56\") "
Nov 02 18:12:45.910620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.910366    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/714ec0a1-9652-4524-81d9-1ca9dc846a56-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714ec0a1-9652-4524-81d9-1ca9dc846a56-maas-api-xks-ca\") pod \"714ec0a1-9652-4524-81d9-1ca9dc846a56\" (UID: \"714ec0a1-9652-4524-81d9-1ca9dc846a56\") "
Nov 02 18:12:45.910620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.910450    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/714ec0a1-9652-4524-81d9-1ca9dc846a56-maas-api-openshift-ca" pod "714ec0a1-9652-4524-81d9-1ca9dc846a56" (UID: "714ec0a1-9652-4524-81d9-1ca9dc846a56"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:45.910792 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.910773    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/714ec0a1-9652-4524-81d9-1ca9dc846a56-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:45.912510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.912482    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/714ec0a1-9652-4524-81d9-1ca9dc846a56-kube-api-access-9bzq6" pod "714ec0a1-9652-4524-81d9-1ca9dc846a56" (UID: "714ec0a1-9652-4524-81d9-1ca9dc846a56"). InnerVolumeSpecName "kube-api-access-9bzq6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:45.912588 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:45.912557    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/714ec0a1-9652-4524-81d9-1ca9dc846a56-maas-api-xks-ca" pod "714ec0a1-9652-4524-81d9-1ca9dc846a56" (UID: "714ec0a1-9652-4524-81d9-1ca9dc846a56"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:46.012340 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.012295    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-9bzq6\" (UniqueName: \"kubernetes.io/projected/714ec0a1-9652-4524-81d9-1ca9dc846a56-kube-api-access-9bzq6\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:46.012340 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.012334    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/714ec0a1-9652-4524-81d9-1ca9dc846a56-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:46.120137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.120102    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:46.166027 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:46.165947    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:12:46.279603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.279563    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:46.426530 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.426386    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:46.594937 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.594911    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"
Nov 02 18:12:46.595103 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.594956    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m" event={"ID":"714ec0a1-9652-4524-81d9-1ca9dc846a56","Type":"ContainerDied","Data":"c153ce64ebce801d160b4539afd375a7c2872c04fb7b7b2620f918544d280bb8"}
Nov 02 18:12:46.595103 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.595015    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c153ce64ebce801d160b4539afd375a7c2872c04fb7b7b2620f918544d280bb8"
Nov 02 18:12:46.597308 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.597285    4456 generic.go:356] "Generic (PLEG): container finished" podID="acd44782-84e1-411a-9a1c-6d21456733bb" containerID="5d1811fa76419aa50beae48550ad5ab8de1c9137d28d84239f25338279ddf29d" exitCode=0
Nov 02 18:12:46.597442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.597350    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" event={"ID":"acd44782-84e1-411a-9a1c-6d21456733bb","Type":"ContainerDied","Data":"5d1811fa76419aa50beae48550ad5ab8de1c9137d28d84239f25338279ddf29d"}
Nov 02 18:12:46.597442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.597407    4456 scope.go:122] "RemoveContainer" containerID="c07d4b5f3cb6c757a1c1c8c0ec52b7bca550932f78aa3af4048216e01b4efc76"
Nov 02 18:12:46.748228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.748186    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 18:12:46.791776 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.791749    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"
Nov 02 18:12:46.929580 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.929545    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-credential-socket\") pod \"acd44782-84e1-411a-9a1c-6d21456733bb\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") "
Nov 02 18:12:46.930003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.929614    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-kube-api-access-fdp88\" (UniqueName: \"kubernetes.io/projected/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-kube-api-access-fdp88\") pod \"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4\" (UID: \"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4\") "
Nov 02 18:12:46.930003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.929640    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-istio-data\") pod \"acd44782-84e1-411a-9a1c-6d21456733bb\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") "
Nov 02 18:12:46.930003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.929657    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd44782-84e1-411a-9a1c-6d21456733bb-istio-token\" (UniqueName: \"kubernetes.io/projected/acd44782-84e1-411a-9a1c-6d21456733bb-istio-token\") pod \"acd44782-84e1-411a-9a1c-6d21456733bb\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") "
Nov 02 18:12:46.930003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.929692    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-workload-certs\") pod \"acd44782-84e1-411a-9a1c-6d21456733bb\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") "
Nov 02 18:12:46.930003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.929773    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/acd44782-84e1-411a-9a1c-6d21456733bb-kube-api-access-k7556\" (UniqueName: \"kubernetes.io/projected/acd44782-84e1-411a-9a1c-6d21456733bb-kube-api-access-k7556\") pod \"acd44782-84e1-411a-9a1c-6d21456733bb\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") "
Nov 02 18:12:46.930285 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.929857    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-workload-socket\") pod \"acd44782-84e1-411a-9a1c-6d21456733bb\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") "
Nov 02 18:12:46.930285 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.929890    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/acd44782-84e1-411a-9a1c-6d21456733bb-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd44782-84e1-411a-9a1c-6d21456733bb-istio-podinfo\") pod \"acd44782-84e1-411a-9a1c-6d21456733bb\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") "
Nov 02 18:12:46.930285 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.929917    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-maas-api-openshift-ca\") pod \"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4\" (UID: \"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4\") "
Nov 02 18:12:46.930285 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.929946    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-istio-data" pod "acd44782-84e1-411a-9a1c-6d21456733bb" (UID: "acd44782-84e1-411a-9a1c-6d21456733bb"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:46.930285 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.929957    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-credential-socket" pod "acd44782-84e1-411a-9a1c-6d21456733bb" (UID: "acd44782-84e1-411a-9a1c-6d21456733bb"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:46.930535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.929976    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/acd44782-84e1-411a-9a1c-6d21456733bb-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd44782-84e1-411a-9a1c-6d21456733bb-istiod-ca-cert\") pod \"acd44782-84e1-411a-9a1c-6d21456733bb\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") "
Nov 02 18:12:46.930535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.930014    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-maas-api-xks-ca\") pod \"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4\" (UID: \"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4\") "
Nov 02 18:12:46.930535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.930088    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-istio-envoy\") pod \"acd44782-84e1-411a-9a1c-6d21456733bb\" (UID: \"acd44782-84e1-411a-9a1c-6d21456733bb\") "
Nov 02 18:12:46.930535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.930136    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-workload-certs" pod "acd44782-84e1-411a-9a1c-6d21456733bb" (UID: "acd44782-84e1-411a-9a1c-6d21456733bb"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:46.930535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.930348    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-maas-api-openshift-ca" pod "7f3984cc-87b7-4b39-9a93-d0f0961fc5d4" (UID: "7f3984cc-87b7-4b39-9a93-d0f0961fc5d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:46.930781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.930364    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-workload-socket" pod "acd44782-84e1-411a-9a1c-6d21456733bb" (UID: "acd44782-84e1-411a-9a1c-6d21456733bb"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:46.930781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.930629    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:46.930781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.930654    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:46.930781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.930668    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:46.930781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.930681    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:46.930781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.930693    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:46.931525 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.931487    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/acd44782-84e1-411a-9a1c-6d21456733bb-istiod-ca-cert" pod "acd44782-84e1-411a-9a1c-6d21456733bb" (UID: "acd44782-84e1-411a-9a1c-6d21456733bb"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:46.932958 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.932867    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-kube-api-access-fdp88" pod "7f3984cc-87b7-4b39-9a93-d0f0961fc5d4" (UID: "7f3984cc-87b7-4b39-9a93-d0f0961fc5d4"). InnerVolumeSpecName "kube-api-access-fdp88". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:46.932958 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.932891    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd44782-84e1-411a-9a1c-6d21456733bb-kube-api-access-k7556" pod "acd44782-84e1-411a-9a1c-6d21456733bb" (UID: "acd44782-84e1-411a-9a1c-6d21456733bb"). InnerVolumeSpecName "kube-api-access-k7556". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:46.933160 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.933140    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-maas-api-xks-ca" pod "7f3984cc-87b7-4b39-9a93-d0f0961fc5d4" (UID: "7f3984cc-87b7-4b39-9a93-d0f0961fc5d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:46.933308 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.933288    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/acd44782-84e1-411a-9a1c-6d21456733bb-istio-token" pod "acd44782-84e1-411a-9a1c-6d21456733bb" (UID: "acd44782-84e1-411a-9a1c-6d21456733bb"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:46.933488 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.933468    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/acd44782-84e1-411a-9a1c-6d21456733bb-istio-podinfo" pod "acd44782-84e1-411a-9a1c-6d21456733bb" (UID: "acd44782-84e1-411a-9a1c-6d21456733bb"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:12:46.933715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:46.933697    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-istio-envoy" pod "acd44782-84e1-411a-9a1c-6d21456733bb" (UID: "acd44782-84e1-411a-9a1c-6d21456733bb"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:47.031807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.031735    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fdp88\" (UniqueName: \"kubernetes.io/projected/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-kube-api-access-fdp88\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:47.031807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.031766    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/acd44782-84e1-411a-9a1c-6d21456733bb-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:47.031807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.031776    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-k7556\" (UniqueName: \"kubernetes.io/projected/acd44782-84e1-411a-9a1c-6d21456733bb-kube-api-access-k7556\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:47.031807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.031785    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/acd44782-84e1-411a-9a1c-6d21456733bb-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:47.031807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.031793    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/acd44782-84e1-411a-9a1c-6d21456733bb-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:47.031807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.031801    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:47.031807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.031809    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/acd44782-84e1-411a-9a1c-6d21456733bb-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:47.130350 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.130301    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"]
Nov 02 18:12:47.130795 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.130779    4456 cpu_manager.go:414] "containerMap: removing container" podUID="7f3984cc-87b7-4b39-9a93-d0f0961fc5d4" containerName="revoke-keys"
Nov 02 18:12:47.130795 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.130794    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f3984cc-87b7-4b39-9a93-d0f0961fc5d4" containerName="revoke-keys"
Nov 02 18:12:47.130963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.130807    4456 cpu_manager.go:414] "containerMap: removing container" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy"
Nov 02 18:12:47.130963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.130829    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy"
Nov 02 18:12:47.130963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.130839    4456 cpu_manager.go:414] "containerMap: removing container" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy"
Nov 02 18:12:47.130963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.130847    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy"
Nov 02 18:12:47.130963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.130891    4456 cpu_manager.go:414] "containerMap: removing container" podUID="2e0ba0a6-de17-49db-91c5-4d4fe845aa67" containerName="revoke-keys"
Nov 02 18:12:47.130963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.130895    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2e0ba0a6-de17-49db-91c5-4d4fe845aa67" containerName="revoke-keys"
Nov 02 18:12:47.130963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.130909    4456 cpu_manager.go:414] "containerMap: removing container" podUID="714ec0a1-9652-4524-81d9-1ca9dc846a56" containerName="revoke-keys"
Nov 02 18:12:47.130963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.130913    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="714ec0a1-9652-4524-81d9-1ca9dc846a56" containerName="revoke-keys"
Nov 02 18:12:47.131339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.130985    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="714ec0a1-9652-4524-81d9-1ca9dc846a56" containerName="revoke-keys"
Nov 02 18:12:47.131339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.130996    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f3984cc-87b7-4b39-9a93-d0f0961fc5d4" containerName="revoke-keys"
Nov 02 18:12:47.131339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.131002    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy"
Nov 02 18:12:47.131339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.131007    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" containerName="istio-proxy"
Nov 02 18:12:47.131339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.131020    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="2e0ba0a6-de17-49db-91c5-4d4fe845aa67" containerName="revoke-keys"
Nov 02 18:12:47.135788 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.135761    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.155169 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.155125    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-shared-a-w0-f91b2d-openshift-default-dockercfg-st4mz\""
Nov 02 18:12:47.160929 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.160903    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"]
Nov 02 18:12:47.235186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.235154    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-credential-socket\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.235361 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.235220    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f9278f-4f06-4809-9594-cd1b09070c5c-istiod-ca-cert\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.235361 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.235342    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-workload-socket\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.235447 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.235392    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-podinfo\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.235447 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.235410    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-token\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.235447 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.235429    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-29dwk\" (UniqueName: \"kubernetes.io/projected/56f9278f-4f06-4809-9594-cd1b09070c5c-kube-api-access-29dwk\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.235536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.235498    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-data\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.235536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.235531    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-workload-certs\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.235596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.235556    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-envoy\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.336909 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.336827    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-podinfo\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.336909 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.336871    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-token\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.336909 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.336894    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-29dwk\" (UniqueName: \"kubernetes.io/projected/56f9278f-4f06-4809-9594-cd1b09070c5c-kube-api-access-29dwk\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.337180 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.336921    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-data\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.337180 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.336974    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-workload-certs\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.337180 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.337012    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-envoy\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.337180 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.337097    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-credential-socket\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.337358 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.337186    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f9278f-4f06-4809-9594-cd1b09070c5c-istiod-ca-cert\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.337358 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.337243    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-workload-socket\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.337464 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.337413    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-workload-certs\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.337512 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.337473    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-data\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.337622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.337596    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-credential-socket\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.337944 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.337673    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-workload-socket\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.338035 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.338008    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f9278f-4f06-4809-9594-cd1b09070c5c-istiod-ca-cert\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.339468 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.339442    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-envoy\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.339799 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.339783    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-podinfo\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.361846 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.361799    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-token\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.361934 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.361917    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-29dwk\" (UniqueName: \"kubernetes.io/projected/56f9278f-4f06-4809-9594-cd1b09070c5c-kube-api-access-29dwk\") pod \"e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") " pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.454467 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.454428    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:47.606577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.606495    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"
Nov 02 18:12:47.609028 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.609001    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"
Nov 02 18:12:47.620530 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.620486    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg" event={"ID":"7f3984cc-87b7-4b39-9a93-d0f0961fc5d4","Type":"ContainerDied","Data":"b8807c56ebd83df739bde06828872d77101c778cdfff98f737e46da046826d36"}
Nov 02 18:12:47.620530 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.620525    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b8807c56ebd83df739bde06828872d77101c778cdfff98f737e46da046826d36"
Nov 02 18:12:47.620710 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.620539    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq" event={"ID":"acd44782-84e1-411a-9a1c-6d21456733bb","Type":"ContainerDied","Data":"20ccacdbcdc8a5973d24d7c2aeedd34afed8581ce29954a8d6e0374ab30d9bf1"}
Nov 02 18:12:47.620710 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.620590    4456 scope.go:122] "RemoveContainer" containerID="5d1811fa76419aa50beae48550ad5ab8de1c9137d28d84239f25338279ddf29d"
Nov 02 18:12:47.668103 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.668068    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:47.690156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.690113    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"]
Nov 02 18:12:47.693483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.693452    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w1-120394-openshift-default-6459698f85-j24dq"]
Nov 02 18:12:47.777123 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:12:47.777093    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod56f9278f_4f06_4809_9594_cd1b09070c5c.slice/crio-25ae22d3a71f0b89370d8fe11d16125c659d9edd89c27c1f618dd98670977f95 WatchSource:0}: Error finding container 25ae22d3a71f0b89370d8fe11d16125c659d9edd89c27c1f618dd98670977f95: Status 404 returned error can't find the container with id 25ae22d3a71f0b89370d8fe11d16125c659d9edd89c27c1f618dd98670977f95
Nov 02 18:12:47.777868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.777846    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"]
Nov 02 18:12:47.780182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.780144    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:12:47.780267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.780231    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:12:47.780311 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.780276    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:12:47.865960 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:47.865914    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:12:48.046632 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.046591    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-credential-socket\") pod \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") "
Nov 02 18:12:48.047150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.046665    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-token\" (UniqueName: \"kubernetes.io/projected/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-token\") pod \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") "
Nov 02 18:12:48.047150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.046706    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-podinfo\") pod \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") "
Nov 02 18:12:48.047150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.046739    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-workload-certs\") pod \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") "
Nov 02 18:12:48.047150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.046838    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/856c2dca-bad2-4f62-a536-1ca0e09f0a12-kube-api-access-shksx\" (UniqueName: \"kubernetes.io/projected/856c2dca-bad2-4f62-a536-1ca0e09f0a12-kube-api-access-shksx\") pod \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") "
Nov 02 18:12:48.047150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.046896    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-data\") pod \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") "
Nov 02 18:12:48.047403 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.046939    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-workload-socket\") pod \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") "
Nov 02 18:12:48.047767 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.047553    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-envoy\") pod \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") "
Nov 02 18:12:48.047767 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.047639    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istiod-ca-cert\") pod \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\" (UID: \"856c2dca-bad2-4f62-a536-1ca0e09f0a12\") "
Nov 02 18:12:48.047977 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.047011    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-credential-socket" pod "856c2dca-bad2-4f62-a536-1ca0e09f0a12" (UID: "856c2dca-bad2-4f62-a536-1ca0e09f0a12"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:48.047977 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.047244    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-workload-certs" pod "856c2dca-bad2-4f62-a536-1ca0e09f0a12" (UID: "856c2dca-bad2-4f62-a536-1ca0e09f0a12"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:48.047977 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.047408    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-workload-socket" pod "856c2dca-bad2-4f62-a536-1ca0e09f0a12" (UID: "856c2dca-bad2-4f62-a536-1ca0e09f0a12"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:48.047977 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.047454    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-data" pod "856c2dca-bad2-4f62-a536-1ca0e09f0a12" (UID: "856c2dca-bad2-4f62-a536-1ca0e09f0a12"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:48.048310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.048287    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:48.048386 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.048317    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:48.048386 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.048322    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istiod-ca-cert" pod "856c2dca-bad2-4f62-a536-1ca0e09f0a12" (UID: "856c2dca-bad2-4f62-a536-1ca0e09f0a12"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:48.048386 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.048330    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:48.048386 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.048378    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:48.050136 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.050096    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856c2dca-bad2-4f62-a536-1ca0e09f0a12-kube-api-access-shksx" pod "856c2dca-bad2-4f62-a536-1ca0e09f0a12" (UID: "856c2dca-bad2-4f62-a536-1ca0e09f0a12"). InnerVolumeSpecName "kube-api-access-shksx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:48.050239 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.050133    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-podinfo" pod "856c2dca-bad2-4f62-a536-1ca0e09f0a12" (UID: "856c2dca-bad2-4f62-a536-1ca0e09f0a12"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:12:48.050685 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.050665    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-envoy" pod "856c2dca-bad2-4f62-a536-1ca0e09f0a12" (UID: "856c2dca-bad2-4f62-a536-1ca0e09f0a12"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:12:48.050979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.050956    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-token" pod "856c2dca-bad2-4f62-a536-1ca0e09f0a12" (UID: "856c2dca-bad2-4f62-a536-1ca0e09f0a12"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:48.149339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.149246    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:48.149339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.149287    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:48.149339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.149299    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:48.149339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.149307    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-shksx\" (UniqueName: \"kubernetes.io/projected/856c2dca-bad2-4f62-a536-1ca0e09f0a12-kube-api-access-shksx\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:48.149339 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.149316    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/856c2dca-bad2-4f62-a536-1ca0e09f0a12-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:48.619648 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.619605    4456 generic.go:356] "Generic (PLEG): container finished" podID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerID="43b444ecc72b8f3eef31fafa69b45fd05eaa0fc026c197971824e9e9553d8fe9" exitCode=0
Nov 02 18:12:48.619881 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.619672    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"
Nov 02 18:12:48.619881 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.619699    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" event={"ID":"856c2dca-bad2-4f62-a536-1ca0e09f0a12","Type":"ContainerDied","Data":"43b444ecc72b8f3eef31fafa69b45fd05eaa0fc026c197971824e9e9553d8fe9"}
Nov 02 18:12:48.619881 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.619857    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt" event={"ID":"856c2dca-bad2-4f62-a536-1ca0e09f0a12","Type":"ContainerDied","Data":"9b54f67d672baa65cc06b469385db955c33c287dcbfc717a6514b0c7eda79d35"}
Nov 02 18:12:48.620072 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.619895    4456 scope.go:122] "RemoveContainer" containerID="43b444ecc72b8f3eef31fafa69b45fd05eaa0fc026c197971824e9e9553d8fe9"
Nov 02 18:12:48.621614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.621585    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p" event={"ID":"56f9278f-4f06-4809-9594-cd1b09070c5c","Type":"ContainerStarted","Data":"3ac6971e7f9a60a87e7a4ec8c4b7e680f887de253334c6c9abaa27988043685b"}
Nov 02 18:12:48.621728 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.621633    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p" event={"ID":"56f9278f-4f06-4809-9594-cd1b09070c5c","Type":"ContainerStarted","Data":"25ae22d3a71f0b89370d8fe11d16125c659d9edd89c27c1f618dd98670977f95"}
Nov 02 18:12:48.631949 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.631763    4456 scope.go:122] "RemoveContainer" containerID="51e90f82024911642a387ee990edbbcce91cbeeb7879f1ae0c57ac3433a8cfc7"
Nov 02 18:12:48.644178 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.644155    4456 scope.go:122] "RemoveContainer" containerID="43b444ecc72b8f3eef31fafa69b45fd05eaa0fc026c197971824e9e9553d8fe9"
Nov 02 18:12:48.645629 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:48.645599    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"43b444ecc72b8f3eef31fafa69b45fd05eaa0fc026c197971824e9e9553d8fe9\": container with ID starting with 43b444ecc72b8f3eef31fafa69b45fd05eaa0fc026c197971824e9e9553d8fe9 not found: ID does not exist" containerID="43b444ecc72b8f3eef31fafa69b45fd05eaa0fc026c197971824e9e9553d8fe9"
Nov 02 18:12:48.645740 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.645635    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"43b444ecc72b8f3eef31fafa69b45fd05eaa0fc026c197971824e9e9553d8fe9"} err="failed to get container status \"43b444ecc72b8f3eef31fafa69b45fd05eaa0fc026c197971824e9e9553d8fe9\": rpc error: code = NotFound desc = could not find container \"43b444ecc72b8f3eef31fafa69b45fd05eaa0fc026c197971824e9e9553d8fe9\": container with ID starting with 43b444ecc72b8f3eef31fafa69b45fd05eaa0fc026c197971824e9e9553d8fe9 not found: ID does not exist"
Nov 02 18:12:48.645740 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.645658    4456 scope.go:122] "RemoveContainer" containerID="51e90f82024911642a387ee990edbbcce91cbeeb7879f1ae0c57ac3433a8cfc7"
Nov 02 18:12:48.646327 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:48.646300    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"51e90f82024911642a387ee990edbbcce91cbeeb7879f1ae0c57ac3433a8cfc7\": container with ID starting with 51e90f82024911642a387ee990edbbcce91cbeeb7879f1ae0c57ac3433a8cfc7 not found: ID does not exist" containerID="51e90f82024911642a387ee990edbbcce91cbeeb7879f1ae0c57ac3433a8cfc7"
Nov 02 18:12:48.646383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.646326    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"51e90f82024911642a387ee990edbbcce91cbeeb7879f1ae0c57ac3433a8cfc7"} err="failed to get container status \"51e90f82024911642a387ee990edbbcce91cbeeb7879f1ae0c57ac3433a8cfc7\": rpc error: code = NotFound desc = could not find container \"51e90f82024911642a387ee990edbbcce91cbeeb7879f1ae0c57ac3433a8cfc7\": container with ID starting with 51e90f82024911642a387ee990edbbcce91cbeeb7879f1ae0c57ac3433a8cfc7 not found: ID does not exist"
Nov 02 18:12:48.708935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.708873    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p" podStartSLOduration=1.7088548019999998 podStartE2EDuration="1.708854802s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:12:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:12:48.699969426 +0000 UTC m=+2585.191032608" watchObservedRunningTime="2026-11-02 18:12:48.708854802 +0000 UTC m=+2585.199917963"
Nov 02 18:12:48.741265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.741226    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"]
Nov 02 18:12:48.746882 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:48.746850    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w3-557feb-openshift-default-784b765bd9-vxnbt"]
Nov 02 18:12:49.455424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:49.455392    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:49.457058 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:49.457024    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 02 18:12:49.457184 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:49.457090    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p" podUID="56f9278f-4f06-4809-9594-cd1b09070c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 02 18:12:49.509842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:49.509538    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:49.618058 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:49.618016    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" path="/var/lib/kubelet/pods/856c2dca-bad2-4f62-a536-1ca0e09f0a12/volumes"
Nov 02 18:12:49.618555 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:49.618539    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="acd44782-84e1-411a-9a1c-6d21456733bb" path="/var/lib/kubelet/pods/acd44782-84e1-411a-9a1c-6d21456733bb/volumes"
Nov 02 18:12:50.455621 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:50.455590    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 02 18:12:50.456003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:50.455642    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p" podUID="56f9278f-4f06-4809-9594-cd1b09070c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 02 18:12:51.277605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:51.277562    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:51.455597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:51.455568    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 02 18:12:51.455740 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:51.455618    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p" podUID="56f9278f-4f06-4809-9594-cd1b09070c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 02 18:12:52.134364 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:52.134319    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:52.278585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:52.278546    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:52.455647 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:52.455614    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 02 18:12:52.455892 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:52.455678    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p" podUID="56f9278f-4f06-4809-9594-cd1b09070c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 02 18:12:53.035224 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:53.035194    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:53.202357 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:53.202313    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:53.435632 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:53.435592    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:53.454969 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:53.454942    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 02 18:12:53.455061 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:53.454990    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p" podUID="56f9278f-4f06-4809-9594-cd1b09070c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 02 18:12:53.597839 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:53.589099    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:54.459962 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.459921    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:54.460322 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.460295    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:54.461032 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.461015    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:12:54.591701 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.588751    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"]
Nov 02 18:12:54.591701 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.589011    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api" containerID="cri-o://0acda9a9044766402073a7c3d99977dbfc769a9c9271311f9a6c2b863409852a" gracePeriod=30
Nov 02 18:12:54.691121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.691084    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:54.861538 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.861506    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"]
Nov 02 18:12:54.878408 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.878377    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 18:12:54.957678 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.957634    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6ebd3467-841a-4331-b149-8acf6e60c12c-kube-api-access-64jnz\" (UniqueName: \"kubernetes.io/projected/6ebd3467-841a-4331-b149-8acf6e60c12c-kube-api-access-64jnz\") pod \"6ebd3467-841a-4331-b149-8acf6e60c12c\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") "
Nov 02 18:12:54.957884 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.957765    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ebd3467-841a-4331-b149-8acf6e60c12c-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebd3467-841a-4331-b149-8acf6e60c12c-maas-api-tls\") pod \"6ebd3467-841a-4331-b149-8acf6e60c12c\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") "
Nov 02 18:12:54.958468 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.958441    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6ebd3467-841a-4331-b149-8acf6e60c12c-metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ebd3467-841a-4331-b149-8acf6e60c12c-metrics-certs\") pod \"6ebd3467-841a-4331-b149-8acf6e60c12c\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") "
Nov 02 18:12:54.958664 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.958613    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6ebd3467-841a-4331-b149-8acf6e60c12c-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ebd3467-841a-4331-b149-8acf6e60c12c-openshift-service-ca\") pod \"6ebd3467-841a-4331-b149-8acf6e60c12c\" (UID: \"6ebd3467-841a-4331-b149-8acf6e60c12c\") "
Nov 02 18:12:54.959046 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.959026    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6ebd3467-841a-4331-b149-8acf6e60c12c-openshift-service-ca" pod "6ebd3467-841a-4331-b149-8acf6e60c12c" (UID: "6ebd3467-841a-4331-b149-8acf6e60c12c"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:12:54.960175 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.960144    4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/6ebd3467-841a-4331-b149-8acf6e60c12c-openshift-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:54.976888 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.966478    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebd3467-841a-4331-b149-8acf6e60c12c-maas-api-tls" pod "6ebd3467-841a-4331-b149-8acf6e60c12c" (UID: "6ebd3467-841a-4331-b149-8acf6e60c12c"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:54.976888 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.970980    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6ebd3467-841a-4331-b149-8acf6e60c12c-metrics-certs" pod "6ebd3467-841a-4331-b149-8acf6e60c12c" (UID: "6ebd3467-841a-4331-b149-8acf6e60c12c"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:12:54.976888 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:54.971094    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6ebd3467-841a-4331-b149-8acf6e60c12c-kube-api-access-64jnz" pod "6ebd3467-841a-4331-b149-8acf6e60c12c" (UID: "6ebd3467-841a-4331-b149-8acf6e60c12c"). InnerVolumeSpecName "kube-api-access-64jnz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:12:55.018613 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.018580    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:55.061734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.061701    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/6ebd3467-841a-4331-b149-8acf6e60c12c-maas-api-tls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:55.061734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.061728    4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/6ebd3467-841a-4331-b149-8acf6e60c12c-metrics-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:55.061734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.061737    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-64jnz\" (UniqueName: \"kubernetes.io/projected/6ebd3467-841a-4331-b149-8acf6e60c12c-kube-api-access-64jnz\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:12:55.165065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.165030    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:55.310011 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.309920    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:55.324241 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.324191    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"]
Nov 02 18:12:55.327193 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.327167    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w2-f0a3b8-v875q"]
Nov 02 18:12:55.458465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.458419    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:55.616096 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.616009    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="30ef929e-6a26-44e0-a4d6-f376244672a8" path="/var/lib/kubelet/pods/30ef929e-6a26-44e0-a4d6-f376244672a8/volumes"
Nov 02 18:12:55.673879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.673809    4456 generic.go:356] "Generic (PLEG): container finished" podID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerID="0acda9a9044766402073a7c3d99977dbfc769a9c9271311f9a6c2b863409852a" exitCode=0
Nov 02 18:12:55.673879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.673881    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"
Nov 02 18:12:55.674063 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.673921    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" event={"ID":"6ebd3467-841a-4331-b149-8acf6e60c12c","Type":"ContainerDied","Data":"0acda9a9044766402073a7c3d99977dbfc769a9c9271311f9a6c2b863409852a"}
Nov 02 18:12:55.674063 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.673977    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v" event={"ID":"6ebd3467-841a-4331-b149-8acf6e60c12c","Type":"ContainerDied","Data":"3effdad3d073c6daba1da297c159b41a5a9f572263586cea3824001afa765a92"}
Nov 02 18:12:55.674063 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.674008    4456 scope.go:122] "RemoveContainer" containerID="0acda9a9044766402073a7c3d99977dbfc769a9c9271311f9a6c2b863409852a"
Nov 02 18:12:55.685538 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.685517    4456 scope.go:122] "RemoveContainer" containerID="e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317"
Nov 02 18:12:55.695538 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.695515    4456 scope.go:122] "RemoveContainer" containerID="0acda9a9044766402073a7c3d99977dbfc769a9c9271311f9a6c2b863409852a"
Nov 02 18:12:55.695941 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:55.695922    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"0acda9a9044766402073a7c3d99977dbfc769a9c9271311f9a6c2b863409852a\": container with ID starting with 0acda9a9044766402073a7c3d99977dbfc769a9c9271311f9a6c2b863409852a not found: ID does not exist" containerID="0acda9a9044766402073a7c3d99977dbfc769a9c9271311f9a6c2b863409852a"
Nov 02 18:12:55.695988 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.695951    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"0acda9a9044766402073a7c3d99977dbfc769a9c9271311f9a6c2b863409852a"} err="failed to get container status \"0acda9a9044766402073a7c3d99977dbfc769a9c9271311f9a6c2b863409852a\": rpc error: code = NotFound desc = could not find container \"0acda9a9044766402073a7c3d99977dbfc769a9c9271311f9a6c2b863409852a\": container with ID starting with 0acda9a9044766402073a7c3d99977dbfc769a9c9271311f9a6c2b863409852a not found: ID does not exist"
Nov 02 18:12:55.695988 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.695970    4456 scope.go:122] "RemoveContainer" containerID="e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317"
Nov 02 18:12:55.696272 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:12:55.696246    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317\": container with ID starting with e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317 not found: ID does not exist" containerID="e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317"
Nov 02 18:12:55.696357 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.696276    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317"} err="failed to get container status \"e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317\": rpc error: code = NotFound desc = could not find container \"e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317\": container with ID starting with e92c75b77af38eb972ff44ee705f577615c139ff34a72edcbb9971fd4875c317 not found: ID does not exist"
Nov 02 18:12:55.736099 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.736061    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"]
Nov 02 18:12:55.740260 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.740233    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-w2-f0a3b8-b5cd965b7-p8m8v"]
Nov 02 18:12:55.823857 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:55.823803    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:56.987447 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:56.987413    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:57.544552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:57.544518    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:57.623140 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:57.617339    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" path="/var/lib/kubelet/pods/6ebd3467-841a-4331-b149-8acf6e60c12c/volumes"
Nov 02 18:12:57.692371 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:57.692329    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:57.694276 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:57.694255    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"]
Nov 02 18:12:57.706636 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:57.706607    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w6-f6da5f-2cz9w"]
Nov 02 18:12:57.839275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:57.839189    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:58.293794 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:58.293759    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:59.556447 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:59.556405    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:12:59.614461 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:59.614422    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="72eb6022-bf9b-43da-933a-b6fae2f5c142" path="/var/lib/kubelet/pods/72eb6022-bf9b-43da-933a-b6fae2f5c142/volumes"
Nov 02 18:12:59.862718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:12:59.862631    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" podUID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" containerName="payload-processing" containerID="cri-o://7178d694cd88c8483d23321e761c3e272f6e8be0c2d9e09eb7e85fce91e1f001" gracePeriod=25
Nov 02 18:13:00.146858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.146829    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"
Nov 02 18:13:00.149318 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.149295    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:00.219643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.219590    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922-kube-api-access-sm4wt\" (UniqueName: \"kubernetes.io/projected/97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922-kube-api-access-sm4wt\") pod \"97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922\" (UID: \"97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922\") "
Nov 02 18:13:00.219865 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.219778    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922-plugins-config-volume\") pod \"97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922\" (UID: \"97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922\") "
Nov 02 18:13:00.220255 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.220224    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922-plugins-config-volume" pod "97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" (UID: "97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:00.220374 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.220321    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:00.222683 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.222656    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922-kube-api-access-sm4wt" pod "97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" (UID: "97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922"). InnerVolumeSpecName "kube-api-access-sm4wt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:00.274861 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.274832    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"]
Nov 02 18:13:00.279533 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.279507    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w3-227939-wc2xg"]
Nov 02 18:13:00.308510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.308471    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:00.321528 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.321496    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sm4wt\" (UniqueName: \"kubernetes.io/projected/97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922-kube-api-access-sm4wt\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:00.720260 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.720226    4456 generic.go:356] "Generic (PLEG): container finished" podID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" containerID="7178d694cd88c8483d23321e761c3e272f6e8be0c2d9e09eb7e85fce91e1f001" exitCode=0
Nov 02 18:13:00.720721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.720284    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"
Nov 02 18:13:00.720721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.720308    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" event={"ID":"97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922","Type":"ContainerDied","Data":"7178d694cd88c8483d23321e761c3e272f6e8be0c2d9e09eb7e85fce91e1f001"}
Nov 02 18:13:00.720721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.720369    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2" event={"ID":"97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922","Type":"ContainerDied","Data":"551d09e335d3cc978253a0c35c85f21c1615ab274de47330246e1f404508b03b"}
Nov 02 18:13:00.720721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.720392    4456 scope.go:122] "RemoveContainer" containerID="7178d694cd88c8483d23321e761c3e272f6e8be0c2d9e09eb7e85fce91e1f001"
Nov 02 18:13:00.733684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.733662    4456 scope.go:122] "RemoveContainer" containerID="7178d694cd88c8483d23321e761c3e272f6e8be0c2d9e09eb7e85fce91e1f001"
Nov 02 18:13:00.734116 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:00.734089    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7178d694cd88c8483d23321e761c3e272f6e8be0c2d9e09eb7e85fce91e1f001\": container with ID starting with 7178d694cd88c8483d23321e761c3e272f6e8be0c2d9e09eb7e85fce91e1f001 not found: ID does not exist" containerID="7178d694cd88c8483d23321e761c3e272f6e8be0c2d9e09eb7e85fce91e1f001"
Nov 02 18:13:00.734203 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.734128    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7178d694cd88c8483d23321e761c3e272f6e8be0c2d9e09eb7e85fce91e1f001"} err="failed to get container status \"7178d694cd88c8483d23321e761c3e272f6e8be0c2d9e09eb7e85fce91e1f001\": rpc error: code = NotFound desc = could not find container \"7178d694cd88c8483d23321e761c3e272f6e8be0c2d9e09eb7e85fce91e1f001\": container with ID starting with 7178d694cd88c8483d23321e761c3e272f6e8be0c2d9e09eb7e85fce91e1f001 not found: ID does not exist"
Nov 02 18:13:00.762026 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.761999    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:00.779846 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.779805    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"]
Nov 02 18:13:00.792729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:00.792695    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w3-227939-8649d7c97c-vhjg2"]
Nov 02 18:13:01.518599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:01.518564    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:01.614084 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:01.614054    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f3984cc-87b7-4b39-9a93-d0f0961fc5d4" path="/var/lib/kubelet/pods/7f3984cc-87b7-4b39-9a93-d0f0961fc5d4/volumes"
Nov 02 18:13:01.614508 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:01.614493    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" path="/var/lib/kubelet/pods/97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922/volumes"
Nov 02 18:13:01.884969 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:01.884874    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"]
Nov 02 18:13:01.885402 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:01.885145    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api" containerID="cri-o://b1443853392ccb3e8ec99e7b4b965424d6ddec7a68fbdb6bf5bf89d1a1f57073" gracePeriod=30
Nov 02 18:13:01.956406 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:01.956371    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:01.965373 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:01.965336    4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7f2d24bd_15b2_4eef_a5fc_5cfc3be9417f.slice/crio-b1443853392ccb3e8ec99e7b4b965424d6ddec7a68fbdb6bf5bf89d1a1f57073.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:13:02.170156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.170117    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 18:13:02.249597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.249564    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-maas-api-tls\") pod \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") "
Nov 02 18:13:02.249773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.249656    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-metrics-certs\") pod \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") "
Nov 02 18:13:02.249773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.249726    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-openshift-service-ca\") pod \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") "
Nov 02 18:13:02.249926 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.249876    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-kube-api-access-hmzvt\" (UniqueName: \"kubernetes.io/projected/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-kube-api-access-hmzvt\") pod \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\" (UID: \"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f\") "
Nov 02 18:13:02.250158 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.250133    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-openshift-service-ca" pod "7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" (UID: "7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:02.250328 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.250309    4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-openshift-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:02.251974 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.251943    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-maas-api-tls" pod "7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" (UID: "7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:02.252443 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.252421    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-metrics-certs" pod "7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" (UID: "7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:02.252650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.252630    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-kube-api-access-hmzvt" pod "7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" (UID: "7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f"). InnerVolumeSpecName "kube-api-access-hmzvt". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:02.351985 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.351946    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-maas-api-tls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:02.351985 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.351983    4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-metrics-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:02.351985 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.351998    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hmzvt\" (UniqueName: \"kubernetes.io/projected/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f-kube-api-access-hmzvt\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:02.549181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.549134    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:02.660244 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.660201    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"]
Nov 02 18:13:02.670979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.670948    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w4-2672ed-pjtqf"]
Nov 02 18:13:02.696886 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.696855    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:02.738659 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.738625    4456 generic.go:356] "Generic (PLEG): container finished" podID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerID="b1443853392ccb3e8ec99e7b4b965424d6ddec7a68fbdb6bf5bf89d1a1f57073" exitCode=0
Nov 02 18:13:02.738854 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.738684    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"
Nov 02 18:13:02.738854 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.738708    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" event={"ID":"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f","Type":"ContainerDied","Data":"b1443853392ccb3e8ec99e7b4b965424d6ddec7a68fbdb6bf5bf89d1a1f57073"}
Nov 02 18:13:02.738854 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.738763    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75" event={"ID":"7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f","Type":"ContainerDied","Data":"cce0ef7eba3b852d6ef66e46afdd1ea0bcf24431ac65845ef1aa97ca7f6135e2"}
Nov 02 18:13:02.738854 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.738787    4456 scope.go:122] "RemoveContainer" containerID="b1443853392ccb3e8ec99e7b4b965424d6ddec7a68fbdb6bf5bf89d1a1f57073"
Nov 02 18:13:02.749777 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.749758    4456 scope.go:122] "RemoveContainer" containerID="b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436"
Nov 02 18:13:02.761243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.761213    4456 scope.go:122] "RemoveContainer" containerID="b1443853392ccb3e8ec99e7b4b965424d6ddec7a68fbdb6bf5bf89d1a1f57073"
Nov 02 18:13:02.761524 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:02.761501    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b1443853392ccb3e8ec99e7b4b965424d6ddec7a68fbdb6bf5bf89d1a1f57073\": container with ID starting with b1443853392ccb3e8ec99e7b4b965424d6ddec7a68fbdb6bf5bf89d1a1f57073 not found: ID does not exist" containerID="b1443853392ccb3e8ec99e7b4b965424d6ddec7a68fbdb6bf5bf89d1a1f57073"
Nov 02 18:13:02.761601 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.761540    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b1443853392ccb3e8ec99e7b4b965424d6ddec7a68fbdb6bf5bf89d1a1f57073"} err="failed to get container status \"b1443853392ccb3e8ec99e7b4b965424d6ddec7a68fbdb6bf5bf89d1a1f57073\": rpc error: code = NotFound desc = could not find container \"b1443853392ccb3e8ec99e7b4b965424d6ddec7a68fbdb6bf5bf89d1a1f57073\": container with ID starting with b1443853392ccb3e8ec99e7b4b965424d6ddec7a68fbdb6bf5bf89d1a1f57073 not found: ID does not exist"
Nov 02 18:13:02.761601 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.761564    4456 scope.go:122] "RemoveContainer" containerID="b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436"
Nov 02 18:13:02.761879 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:02.761860    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436\": container with ID starting with b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436 not found: ID does not exist" containerID="b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436"
Nov 02 18:13:02.761926 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.761886    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436"} err="failed to get container status \"b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436\": rpc error: code = NotFound desc = could not find container \"b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436\": container with ID starting with b60097cc688ed8485a2f4536a45cd385dd6139fd10668a9dfebd8e9ca3f15436 not found: ID does not exist"
Nov 02 18:13:02.815972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.815884    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"]
Nov 02 18:13:02.817519 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.817498    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-shared-b-w4-2672ed-6c4bd6f46f-gmm75"]
Nov 02 18:13:02.922756 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:02.922709    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:03.132852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:03.132756    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:03.613996 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:03.613958    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2e0ba0a6-de17-49db-91c5-4d4fe845aa67" path="/var/lib/kubelet/pods/2e0ba0a6-de17-49db-91c5-4d4fe845aa67/volumes"
Nov 02 18:13:03.614489 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:03.614474    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" path="/var/lib/kubelet/pods/7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f/volumes"
Nov 02 18:13:04.469328 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:04.469295    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:05.011261 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:05.011224    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:05.146763 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:05.146731    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"]
Nov 02 18:13:05.160940 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:05.160910    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w5-c810ca-xxx6m"]
Nov 02 18:13:05.162710 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:05.162678    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:05.324006 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:05.323913    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:05.468017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:05.467984    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:05.615908 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:05.615828    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="714ec0a1-9652-4524-81d9-1ca9dc846a56" path="/var/lib/kubelet/pods/714ec0a1-9652-4524-81d9-1ca9dc846a56/volumes"
Nov 02 18:13:05.685047 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:05.685012    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:05.842028 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:05.841991    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:06.812402 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:06.812371    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:07.122884 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.122779    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:07.283907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.283867    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:07.652321 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.652283    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:07.855889 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.855788    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"]
Nov 02 18:13:07.856287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856273    4456 cpu_manager.go:414] "containerMap: removing container" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api"
Nov 02 18:13:07.856325 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856288    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api"
Nov 02 18:13:07.856325 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856316    4456 cpu_manager.go:414] "containerMap: removing container" podUID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" containerName="payload-processing"
Nov 02 18:13:07.856325 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856323    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" containerName="payload-processing"
Nov 02 18:13:07.856421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856336    4456 cpu_manager.go:414] "containerMap: removing container" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api"
Nov 02 18:13:07.856421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856341    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api"
Nov 02 18:13:07.856421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856348    4456 cpu_manager.go:414] "containerMap: removing container" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api"
Nov 02 18:13:07.856421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856352    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api"
Nov 02 18:13:07.856421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856359    4456 cpu_manager.go:414] "containerMap: removing container" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api"
Nov 02 18:13:07.856421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856363    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api"
Nov 02 18:13:07.856421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856370    4456 cpu_manager.go:414] "containerMap: removing container" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api"
Nov 02 18:13:07.856421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856374    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api"
Nov 02 18:13:07.856421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856380    4456 cpu_manager.go:414] "containerMap: removing container" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy"
Nov 02 18:13:07.856421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856384    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy"
Nov 02 18:13:07.856421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856392    4456 cpu_manager.go:414] "containerMap: removing container" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy"
Nov 02 18:13:07.856421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856396    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy"
Nov 02 18:13:07.856844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856506    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="97d9cb62-c0d1-4fc9-a394-9e5b1c8f4922" containerName="payload-processing"
Nov 02 18:13:07.856844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856517    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api"
Nov 02 18:13:07.856844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856523    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api"
Nov 02 18:13:07.856844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856528    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api"
Nov 02 18:13:07.856844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856534    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api"
Nov 02 18:13:07.856844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856538    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy"
Nov 02 18:13:07.856844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856544    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="856c2dca-bad2-4f62-a536-1ca0e09f0a12" containerName="istio-proxy"
Nov 02 18:13:07.856844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.856550    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api"
Nov 02 18:13:07.861121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.861106    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"
Nov 02 18:13:07.880132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.880101    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"]
Nov 02 18:13:07.889598 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.889557    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:13:07.889598 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.889580    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:13:07.926600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.926561    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:13:07.926600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:07.926591    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:13:08.016066 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:08.016033    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-qh642\" (UniqueName: \"kubernetes.io/projected/ea498d1b-474a-42c1-a576-2e0005a2a8c6-kube-api-access-qh642\") pod \"maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9\" (UID: \"ea498d1b-474a-42c1-a576-2e0005a2a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"
Nov 02 18:13:08.016245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:08.016080    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea498d1b-474a-42c1-a576-2e0005a2a8c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9\" (UID: \"ea498d1b-474a-42c1-a576-2e0005a2a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"
Nov 02 18:13:08.016245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:08.016214    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea498d1b-474a-42c1-a576-2e0005a2a8c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9\" (UID: \"ea498d1b-474a-42c1-a576-2e0005a2a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"
Nov 02 18:13:08.117348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:08.117265    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea498d1b-474a-42c1-a576-2e0005a2a8c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9\" (UID: \"ea498d1b-474a-42c1-a576-2e0005a2a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"
Nov 02 18:13:08.117510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:08.117376    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-qh642\" (UniqueName: \"kubernetes.io/projected/ea498d1b-474a-42c1-a576-2e0005a2a8c6-kube-api-access-qh642\") pod \"maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9\" (UID: \"ea498d1b-474a-42c1-a576-2e0005a2a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"
Nov 02 18:13:08.117510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:08.117411    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea498d1b-474a-42c1-a576-2e0005a2a8c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9\" (UID: \"ea498d1b-474a-42c1-a576-2e0005a2a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"
Nov 02 18:13:08.118188 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:08.118109    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea498d1b-474a-42c1-a576-2e0005a2a8c6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9\" (UID: \"ea498d1b-474a-42c1-a576-2e0005a2a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"
Nov 02 18:13:08.119940 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:08.119914    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea498d1b-474a-42c1-a576-2e0005a2a8c6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9\" (UID: \"ea498d1b-474a-42c1-a576-2e0005a2a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"
Nov 02 18:13:08.141718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:08.141692    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-qh642\" (UniqueName: \"kubernetes.io/projected/ea498d1b-474a-42c1-a576-2e0005a2a8c6-kube-api-access-qh642\") pod \"maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9\" (UID: \"ea498d1b-474a-42c1-a576-2e0005a2a8c6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"
Nov 02 18:13:08.174888 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:08.174864    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"
Nov 02 18:13:08.479397 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:08.479367    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"]
Nov 02 18:13:08.479557 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:13:08.479538    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podea498d1b_474a_42c1_a576_2e0005a2a8c6.slice/crio-5abdcfe991c2143e28f3b708597727d1808f63528a87b991e72c383ca05a418c WatchSource:0}: Error finding container 5abdcfe991c2143e28f3b708597727d1808f63528a87b991e72c383ca05a418c: Status 404 returned error can't find the container with id 5abdcfe991c2143e28f3b708597727d1808f63528a87b991e72c383ca05a418c
Nov 02 18:13:08.753262 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:08.753169    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:08.788577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:08.788538    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9" event={"ID":"ea498d1b-474a-42c1-a576-2e0005a2a8c6","Type":"ContainerStarted","Data":"5abdcfe991c2143e28f3b708597727d1808f63528a87b991e72c383ca05a418c"}
Nov 02 18:13:08.903057 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:08.903031    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:09.141799 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:09.141720    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:09.283869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:09.283803    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:09.569321 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:09.569289    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:09.775204 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:09.775172    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"]
Nov 02 18:13:09.775714 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:09.775694    4456 cpu_manager.go:414] "containerMap: removing container" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api"
Nov 02 18:13:09.775806 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:09.775717    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7f2d24bd-15b2-4eef-a5fc-5cfc3be9417f" containerName="maas-api"
Nov 02 18:13:09.775897 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:09.775885    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6ebd3467-841a-4331-b149-8acf6e60c12c" containerName="maas-api"
Nov 02 18:13:09.779407 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:09.779380    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"
Nov 02 18:13:09.798873 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:09.798806    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"]
Nov 02 18:13:09.800716 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:09.800686    4456 generic.go:356] "Generic (PLEG): container finished" podID="ea498d1b-474a-42c1-a576-2e0005a2a8c6" containerID="bab74c3384f8f0dec76df071459ea93b500da899e848d8bef8c31655c7321583" exitCode=0
Nov 02 18:13:09.800876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:09.800753    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9" event={"ID":"ea498d1b-474a-42c1-a576-2e0005a2a8c6","Type":"ContainerDied","Data":"bab74c3384f8f0dec76df071459ea93b500da899e848d8bef8c31655c7321583"}
Nov 02 18:13:09.938133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:09.938096    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327124e1-a104-41ad-805c-1eb5363445fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7\" (UID: \"327124e1-a104-41ad-805c-1eb5363445fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"
Nov 02 18:13:09.938133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:09.938133    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-5mclg\" (UniqueName: \"kubernetes.io/projected/327124e1-a104-41ad-805c-1eb5363445fa-kube-api-access-5mclg\") pod \"maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7\" (UID: \"327124e1-a104-41ad-805c-1eb5363445fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"
Nov 02 18:13:09.938671 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:09.938252    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327124e1-a104-41ad-805c-1eb5363445fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7\" (UID: \"327124e1-a104-41ad-805c-1eb5363445fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"
Nov 02 18:13:10.039115 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:10.039037    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327124e1-a104-41ad-805c-1eb5363445fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7\" (UID: \"327124e1-a104-41ad-805c-1eb5363445fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"
Nov 02 18:13:10.039115 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:10.039083    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-5mclg\" (UniqueName: \"kubernetes.io/projected/327124e1-a104-41ad-805c-1eb5363445fa-kube-api-access-5mclg\") pod \"maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7\" (UID: \"327124e1-a104-41ad-805c-1eb5363445fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"
Nov 02 18:13:10.039336 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:10.039145    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327124e1-a104-41ad-805c-1eb5363445fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7\" (UID: \"327124e1-a104-41ad-805c-1eb5363445fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"
Nov 02 18:13:10.039861 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:10.039810    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327124e1-a104-41ad-805c-1eb5363445fa-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7\" (UID: \"327124e1-a104-41ad-805c-1eb5363445fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"
Nov 02 18:13:10.041586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:10.041568    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327124e1-a104-41ad-805c-1eb5363445fa-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7\" (UID: \"327124e1-a104-41ad-805c-1eb5363445fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"
Nov 02 18:13:10.064250 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:10.064219    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-5mclg\" (UniqueName: \"kubernetes.io/projected/327124e1-a104-41ad-805c-1eb5363445fa-kube-api-access-5mclg\") pod \"maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7\" (UID: \"327124e1-a104-41ad-805c-1eb5363445fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"
Nov 02 18:13:10.093993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:10.093969    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"
Nov 02 18:13:10.400871 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:10.400841    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"]
Nov 02 18:13:10.401247 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:13:10.401225    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod327124e1_a104_41ad_805c_1eb5363445fa.slice/crio-c068eca5a79458f2886ae10611f5efb89c7415d33d561bd71a89696e9155d88a WatchSource:0}: Error finding container c068eca5a79458f2886ae10611f5efb89c7415d33d561bd71a89696e9155d88a: Status 404 returned error can't find the container with id c068eca5a79458f2886ae10611f5efb89c7415d33d561bd71a89696e9155d88a
Nov 02 18:13:10.809407 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:10.809365    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7" event={"ID":"327124e1-a104-41ad-805c-1eb5363445fa","Type":"ContainerStarted","Data":"3e1f290982f9eb51c9746d0938082f539788e09c6c42daea51dd25c8474dddae"}
Nov 02 18:13:10.809556 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:10.809445    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7" event={"ID":"327124e1-a104-41ad-805c-1eb5363445fa","Type":"ContainerStarted","Data":"c068eca5a79458f2886ae10611f5efb89c7415d33d561bd71a89696e9155d88a"}
Nov 02 18:13:10.852853 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:10.852758    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7" podStartSLOduration=1.8527449740000002 podStartE2EDuration="1.852744974s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:10.851626125 +0000 UTC m=+2607.342689366" watchObservedRunningTime="2026-11-02 18:13:10.852744974 +0000 UTC m=+2607.343808153"
Nov 02 18:13:10.974139 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:10.974114    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"
Nov 02 18:13:11.051781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.051750    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ea498d1b-474a-42c1-a576-2e0005a2a8c6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea498d1b-474a-42c1-a576-2e0005a2a8c6-maas-api-openshift-ca\") pod \"ea498d1b-474a-42c1-a576-2e0005a2a8c6\" (UID: \"ea498d1b-474a-42c1-a576-2e0005a2a8c6\") "
Nov 02 18:13:11.051968 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.051891    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ea498d1b-474a-42c1-a576-2e0005a2a8c6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea498d1b-474a-42c1-a576-2e0005a2a8c6-maas-api-xks-ca\") pod \"ea498d1b-474a-42c1-a576-2e0005a2a8c6\" (UID: \"ea498d1b-474a-42c1-a576-2e0005a2a8c6\") "
Nov 02 18:13:11.052020 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.052010    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ea498d1b-474a-42c1-a576-2e0005a2a8c6-kube-api-access-qh642\" (UniqueName: \"kubernetes.io/projected/ea498d1b-474a-42c1-a576-2e0005a2a8c6-kube-api-access-qh642\") pod \"ea498d1b-474a-42c1-a576-2e0005a2a8c6\" (UID: \"ea498d1b-474a-42c1-a576-2e0005a2a8c6\") "
Nov 02 18:13:11.052437 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.052409    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ea498d1b-474a-42c1-a576-2e0005a2a8c6-maas-api-openshift-ca" pod "ea498d1b-474a-42c1-a576-2e0005a2a8c6" (UID: "ea498d1b-474a-42c1-a576-2e0005a2a8c6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:11.055177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.055154    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ea498d1b-474a-42c1-a576-2e0005a2a8c6-maas-api-xks-ca" pod "ea498d1b-474a-42c1-a576-2e0005a2a8c6" (UID: "ea498d1b-474a-42c1-a576-2e0005a2a8c6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:11.055177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.055166    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ea498d1b-474a-42c1-a576-2e0005a2a8c6-kube-api-access-qh642" pod "ea498d1b-474a-42c1-a576-2e0005a2a8c6" (UID: "ea498d1b-474a-42c1-a576-2e0005a2a8c6"). InnerVolumeSpecName "kube-api-access-qh642". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:11.147396 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.147365    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:11.153867 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.153831    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ea498d1b-474a-42c1-a576-2e0005a2a8c6-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:11.153867 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.153864    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-qh642\" (UniqueName: \"kubernetes.io/projected/ea498d1b-474a-42c1-a576-2e0005a2a8c6-kube-api-access-qh642\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:11.154073 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.153879    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ea498d1b-474a-42c1-a576-2e0005a2a8c6-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:11.437293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.437263    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:11.594856 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.594749    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:11.776587 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.776548    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:11.827849 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.827665    4456 generic.go:356] "Generic (PLEG): container finished" podID="327124e1-a104-41ad-805c-1eb5363445fa" containerID="3e1f290982f9eb51c9746d0938082f539788e09c6c42daea51dd25c8474dddae" exitCode=0
Nov 02 18:13:11.828031 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.827872    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7" event={"ID":"327124e1-a104-41ad-805c-1eb5363445fa","Type":"ContainerDied","Data":"3e1f290982f9eb51c9746d0938082f539788e09c6c42daea51dd25c8474dddae"}
Nov 02 18:13:11.835838 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.830903    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"
Nov 02 18:13:11.835838 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.831647    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9" event={"ID":"ea498d1b-474a-42c1-a576-2e0005a2a8c6","Type":"ContainerDied","Data":"5abdcfe991c2143e28f3b708597727d1808f63528a87b991e72c383ca05a418c"}
Nov 02 18:13:11.835838 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:11.831693    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="5abdcfe991c2143e28f3b708597727d1808f63528a87b991e72c383ca05a418c"
Nov 02 18:13:12.281828 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.281771    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:12.474087 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.474055    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"]
Nov 02 18:13:12.474652 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.474635    4456 cpu_manager.go:414] "containerMap: removing container" podUID="ea498d1b-474a-42c1-a576-2e0005a2a8c6" containerName="revoke-keys"
Nov 02 18:13:12.474697 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.474655    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ea498d1b-474a-42c1-a576-2e0005a2a8c6" containerName="revoke-keys"
Nov 02 18:13:12.474742 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.474734    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ea498d1b-474a-42c1-a576-2e0005a2a8c6" containerName="revoke-keys"
Nov 02 18:13:12.478225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.478206    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"
Nov 02 18:13:12.500512 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.500486    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"]
Nov 02 18:13:12.570331 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.570238    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be3edfab-727f-442d-8fe3-176358694c84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj\" (UID: \"be3edfab-727f-442d-8fe3-176358694c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"
Nov 02 18:13:12.570507 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.570456    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w2cnv\" (UniqueName: \"kubernetes.io/projected/be3edfab-727f-442d-8fe3-176358694c84-kube-api-access-w2cnv\") pod \"maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj\" (UID: \"be3edfab-727f-442d-8fe3-176358694c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"
Nov 02 18:13:12.570507 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.570501    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be3edfab-727f-442d-8fe3-176358694c84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj\" (UID: \"be3edfab-727f-442d-8fe3-176358694c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"
Nov 02 18:13:12.671551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.671520    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w2cnv\" (UniqueName: \"kubernetes.io/projected/be3edfab-727f-442d-8fe3-176358694c84-kube-api-access-w2cnv\") pod \"maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj\" (UID: \"be3edfab-727f-442d-8fe3-176358694c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"
Nov 02 18:13:12.671700 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.671564    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be3edfab-727f-442d-8fe3-176358694c84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj\" (UID: \"be3edfab-727f-442d-8fe3-176358694c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"
Nov 02 18:13:12.671700 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.671638    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be3edfab-727f-442d-8fe3-176358694c84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj\" (UID: \"be3edfab-727f-442d-8fe3-176358694c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"
Nov 02 18:13:12.672304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.672279    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be3edfab-727f-442d-8fe3-176358694c84-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj\" (UID: \"be3edfab-727f-442d-8fe3-176358694c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"
Nov 02 18:13:12.674232 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.674209    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be3edfab-727f-442d-8fe3-176358694c84-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj\" (UID: \"be3edfab-727f-442d-8fe3-176358694c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"
Nov 02 18:13:12.697661 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.697631    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w2cnv\" (UniqueName: \"kubernetes.io/projected/be3edfab-727f-442d-8fe3-176358694c84-kube-api-access-w2cnv\") pod \"maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj\" (UID: \"be3edfab-727f-442d-8fe3-176358694c84\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"
Nov 02 18:13:12.787292 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.787253    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:12.793240 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:12.793210    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"
Nov 02 18:13:13.009572 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.009545    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"
Nov 02 18:13:13.109321 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.109294    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"]
Nov 02 18:13:13.110214 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:13:13.110190    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podbe3edfab_727f_442d_8fe3_176358694c84.slice/crio-f1344fbecb4aab981f20a20c0ce6aff1eb3c331f34605708a1b5a14f4eb09709 WatchSource:0}: Error finding container f1344fbecb4aab981f20a20c0ce6aff1eb3c331f34605708a1b5a14f4eb09709: Status 404 returned error can't find the container with id f1344fbecb4aab981f20a20c0ce6aff1eb3c331f34605708a1b5a14f4eb09709
Nov 02 18:13:13.176350 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.176328    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/327124e1-a104-41ad-805c-1eb5363445fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327124e1-a104-41ad-805c-1eb5363445fa-maas-api-xks-ca\") pod \"327124e1-a104-41ad-805c-1eb5363445fa\" (UID: \"327124e1-a104-41ad-805c-1eb5363445fa\") "
Nov 02 18:13:13.176451 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.176413    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/327124e1-a104-41ad-805c-1eb5363445fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327124e1-a104-41ad-805c-1eb5363445fa-maas-api-openshift-ca\") pod \"327124e1-a104-41ad-805c-1eb5363445fa\" (UID: \"327124e1-a104-41ad-805c-1eb5363445fa\") "
Nov 02 18:13:13.176498 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.176486    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/327124e1-a104-41ad-805c-1eb5363445fa-kube-api-access-5mclg\" (UniqueName: \"kubernetes.io/projected/327124e1-a104-41ad-805c-1eb5363445fa-kube-api-access-5mclg\") pod \"327124e1-a104-41ad-805c-1eb5363445fa\" (UID: \"327124e1-a104-41ad-805c-1eb5363445fa\") "
Nov 02 18:13:13.176745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.176723    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/327124e1-a104-41ad-805c-1eb5363445fa-maas-api-openshift-ca" pod "327124e1-a104-41ad-805c-1eb5363445fa" (UID: "327124e1-a104-41ad-805c-1eb5363445fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:13.176936 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.176920    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/327124e1-a104-41ad-805c-1eb5363445fa-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:13.178597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.178570    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/327124e1-a104-41ad-805c-1eb5363445fa-maas-api-xks-ca" pod "327124e1-a104-41ad-805c-1eb5363445fa" (UID: "327124e1-a104-41ad-805c-1eb5363445fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:13.178680 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.178635    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/327124e1-a104-41ad-805c-1eb5363445fa-kube-api-access-5mclg" pod "327124e1-a104-41ad-805c-1eb5363445fa" (UID: "327124e1-a104-41ad-805c-1eb5363445fa"). InnerVolumeSpecName "kube-api-access-5mclg". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:13.271165 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.271136    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:13.278554 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.278428    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-5mclg\" (UniqueName: \"kubernetes.io/projected/327124e1-a104-41ad-805c-1eb5363445fa-kube-api-access-5mclg\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:13.278554 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.278458    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/327124e1-a104-41ad-805c-1eb5363445fa-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:13.505228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.505185    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:13.650728 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.650646    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:13.842245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.842214    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"]
Nov 02 18:13:13.842469 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.842440    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy" containerID="cri-o://73442968debcfc6a6635d7a8712e48213ab8ebc83bdaf93b458f2f9b8b758f89" gracePeriod=30
Nov 02 18:13:13.849864 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.849803    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj" event={"ID":"be3edfab-727f-442d-8fe3-176358694c84","Type":"ContainerStarted","Data":"256a789ee8ff0b05eeb0b9206df884875f6c586a375dac7962d8407fdab28a9c"}
Nov 02 18:13:13.849990 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.849888    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj" event={"ID":"be3edfab-727f-442d-8fe3-176358694c84","Type":"ContainerStarted","Data":"f1344fbecb4aab981f20a20c0ce6aff1eb3c331f34605708a1b5a14f4eb09709"}
Nov 02 18:13:13.851257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.851237    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"
Nov 02 18:13:13.851346 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.851231    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7" event={"ID":"327124e1-a104-41ad-805c-1eb5363445fa","Type":"ContainerDied","Data":"c068eca5a79458f2886ae10611f5efb89c7415d33d561bd71a89696e9155d88a"}
Nov 02 18:13:13.851382 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.851351    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="c068eca5a79458f2886ae10611f5efb89c7415d33d561bd71a89696e9155d88a"
Nov 02 18:13:13.876193 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:13.876167    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:14.376657 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.376581    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:14.509167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.509100    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj" podStartSLOduration=2.5090800890000002 podStartE2EDuration="2.509080089s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:12 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:13.97140171 +0000 UTC m=+2610.462464889" watchObservedRunningTime="2026-11-02 18:13:14.509080089 +0000 UTC m=+2611.000143251"
Nov 02 18:13:14.510166 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.510143    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"]
Nov 02 18:13:14.518504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.518479    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"]
Nov 02 18:13:14.530504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.530476    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"]
Nov 02 18:13:14.575323 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.575296    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"]
Nov 02 18:13:14.575750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.575739    4456 cpu_manager.go:414] "containerMap: removing container" podUID="327124e1-a104-41ad-805c-1eb5363445fa" containerName="revoke-keys"
Nov 02 18:13:14.575802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.575752    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="327124e1-a104-41ad-805c-1eb5363445fa" containerName="revoke-keys"
Nov 02 18:13:14.575907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.575896    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="327124e1-a104-41ad-805c-1eb5363445fa" containerName="revoke-keys"
Nov 02 18:13:14.578530 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.578514    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"
Nov 02 18:13:14.599129 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.599099    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"]
Nov 02 18:13:14.642114 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.642047    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:14.707714 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.707673    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lhgnj\" (UniqueName: \"kubernetes.io/projected/116e4234-31db-4d7f-b00d-e418850d25b6-kube-api-access-lhgnj\") pod \"maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p\" (UID: \"116e4234-31db-4d7f-b00d-e418850d25b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"
Nov 02 18:13:14.707714 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.707711    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116e4234-31db-4d7f-b00d-e418850d25b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p\" (UID: \"116e4234-31db-4d7f-b00d-e418850d25b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"
Nov 02 18:13:14.708060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.707842    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116e4234-31db-4d7f-b00d-e418850d25b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p\" (UID: \"116e4234-31db-4d7f-b00d-e418850d25b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"
Nov 02 18:13:14.809478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.809441    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lhgnj\" (UniqueName: \"kubernetes.io/projected/116e4234-31db-4d7f-b00d-e418850d25b6-kube-api-access-lhgnj\") pod \"maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p\" (UID: \"116e4234-31db-4d7f-b00d-e418850d25b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"
Nov 02 18:13:14.809669 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.809488    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116e4234-31db-4d7f-b00d-e418850d25b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p\" (UID: \"116e4234-31db-4d7f-b00d-e418850d25b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"
Nov 02 18:13:14.809669 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.809546    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116e4234-31db-4d7f-b00d-e418850d25b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p\" (UID: \"116e4234-31db-4d7f-b00d-e418850d25b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"
Nov 02 18:13:14.810286 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.810256    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116e4234-31db-4d7f-b00d-e418850d25b6-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p\" (UID: \"116e4234-31db-4d7f-b00d-e418850d25b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"
Nov 02 18:13:14.812111 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.812091    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116e4234-31db-4d7f-b00d-e418850d25b6-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p\" (UID: \"116e4234-31db-4d7f-b00d-e418850d25b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"
Nov 02 18:13:14.833832 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.833790    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lhgnj\" (UniqueName: \"kubernetes.io/projected/116e4234-31db-4d7f-b00d-e418850d25b6-kube-api-access-lhgnj\") pod \"maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p\" (UID: \"116e4234-31db-4d7f-b00d-e418850d25b6\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"
Nov 02 18:13:14.860549 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.860517    4456 generic.go:356] "Generic (PLEG): container finished" podID="be3edfab-727f-442d-8fe3-176358694c84" containerID="256a789ee8ff0b05eeb0b9206df884875f6c586a375dac7962d8407fdab28a9c" exitCode=0
Nov 02 18:13:14.860683 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.860566    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj" event={"ID":"be3edfab-727f-442d-8fe3-176358694c84","Type":"ContainerDied","Data":"256a789ee8ff0b05eeb0b9206df884875f6c586a375dac7962d8407fdab28a9c"}
Nov 02 18:13:14.893462 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:14.893383    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"
Nov 02 18:13:15.200549 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:15.200519    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"]
Nov 02 18:13:15.201259 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:13:15.201233    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod116e4234_31db_4d7f_b00d_e418850d25b6.slice/crio-72b2ada97787a5119c8c575f8954d6ad3fc50361b60af8ceeea039d72d5192eb WatchSource:0}: Error finding container 72b2ada97787a5119c8c575f8954d6ad3fc50361b60af8ceeea039d72d5192eb: Status 404 returned error can't find the container with id 72b2ada97787a5119c8c575f8954d6ad3fc50361b60af8ceeea039d72d5192eb
Nov 02 18:13:15.309169 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:15.309133    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:15.869189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:15.869107    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p" event={"ID":"116e4234-31db-4d7f-b00d-e418850d25b6","Type":"ContainerStarted","Data":"c8afec9f578891af7df845dff827ad6f16edb17d077bf006546ff0080c7005a1"}
Nov 02 18:13:15.869189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:15.869164    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p" event={"ID":"116e4234-31db-4d7f-b00d-e418850d25b6","Type":"ContainerStarted","Data":"72b2ada97787a5119c8c575f8954d6ad3fc50361b60af8ceeea039d72d5192eb"}
Nov 02 18:13:15.912348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:15.912289    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p" podStartSLOduration=1.91227189 podStartE2EDuration="1.91227189s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:14 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:15.912009295 +0000 UTC m=+2612.403072484" watchObservedRunningTime="2026-11-02 18:13:15.91227189 +0000 UTC m=+2612.403335051"
Nov 02 18:13:16.017742 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:16.017198    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"
Nov 02 18:13:16.124083 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:16.124004    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/be3edfab-727f-442d-8fe3-176358694c84-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be3edfab-727f-442d-8fe3-176358694c84-maas-api-openshift-ca\") pod \"be3edfab-727f-442d-8fe3-176358694c84\" (UID: \"be3edfab-727f-442d-8fe3-176358694c84\") "
Nov 02 18:13:16.124223 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:16.124193    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/be3edfab-727f-442d-8fe3-176358694c84-kube-api-access-w2cnv\" (UniqueName: \"kubernetes.io/projected/be3edfab-727f-442d-8fe3-176358694c84-kube-api-access-w2cnv\") pod \"be3edfab-727f-442d-8fe3-176358694c84\" (UID: \"be3edfab-727f-442d-8fe3-176358694c84\") "
Nov 02 18:13:16.124266 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:16.124248    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/be3edfab-727f-442d-8fe3-176358694c84-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be3edfab-727f-442d-8fe3-176358694c84-maas-api-xks-ca\") pod \"be3edfab-727f-442d-8fe3-176358694c84\" (UID: \"be3edfab-727f-442d-8fe3-176358694c84\") "
Nov 02 18:13:16.124425 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:16.124406    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/be3edfab-727f-442d-8fe3-176358694c84-maas-api-openshift-ca" pod "be3edfab-727f-442d-8fe3-176358694c84" (UID: "be3edfab-727f-442d-8fe3-176358694c84"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:16.124676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:16.124651    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/be3edfab-727f-442d-8fe3-176358694c84-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:16.126538 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:16.126509    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/be3edfab-727f-442d-8fe3-176358694c84-kube-api-access-w2cnv" pod "be3edfab-727f-442d-8fe3-176358694c84" (UID: "be3edfab-727f-442d-8fe3-176358694c84"). InnerVolumeSpecName "kube-api-access-w2cnv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:16.126662 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:16.126646    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/be3edfab-727f-442d-8fe3-176358694c84-maas-api-xks-ca" pod "be3edfab-727f-442d-8fe3-176358694c84" (UID: "be3edfab-727f-442d-8fe3-176358694c84"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:16.168443 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:16.168406    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:13:16.226218 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:16.226184    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w2cnv\" (UniqueName: \"kubernetes.io/projected/be3edfab-727f-442d-8fe3-176358694c84-kube-api-access-w2cnv\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:16.226218 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:16.226220    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/be3edfab-727f-442d-8fe3-176358694c84-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:16.876520 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:16.876479    4456 generic.go:356] "Generic (PLEG): container finished" podID="116e4234-31db-4d7f-b00d-e418850d25b6" containerID="c8afec9f578891af7df845dff827ad6f16edb17d077bf006546ff0080c7005a1" exitCode=0
Nov 02 18:13:16.877008 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:16.876567    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p" event={"ID":"116e4234-31db-4d7f-b00d-e418850d25b6","Type":"ContainerDied","Data":"c8afec9f578891af7df845dff827ad6f16edb17d077bf006546ff0080c7005a1"}
Nov 02 18:13:16.878123 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:16.878103    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj" event={"ID":"be3edfab-727f-442d-8fe3-176358694c84","Type":"ContainerDied","Data":"f1344fbecb4aab981f20a20c0ce6aff1eb3c331f34605708a1b5a14f4eb09709"}
Nov 02 18:13:16.878195 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:16.878129    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"
Nov 02 18:13:16.878195 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:16.878146    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f1344fbecb4aab981f20a20c0ce6aff1eb3c331f34605708a1b5a14f4eb09709"
Nov 02 18:13:17.178434 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:17.178338    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:17.363803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:17.363773    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:18.033368 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.033337    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"
Nov 02 18:13:18.046035 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.046010    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/116e4234-31db-4d7f-b00d-e418850d25b6-kube-api-access-lhgnj\" (UniqueName: \"kubernetes.io/projected/116e4234-31db-4d7f-b00d-e418850d25b6-kube-api-access-lhgnj\") pod \"116e4234-31db-4d7f-b00d-e418850d25b6\" (UID: \"116e4234-31db-4d7f-b00d-e418850d25b6\") "
Nov 02 18:13:18.046141 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.046093    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/116e4234-31db-4d7f-b00d-e418850d25b6-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116e4234-31db-4d7f-b00d-e418850d25b6-maas-api-openshift-ca\") pod \"116e4234-31db-4d7f-b00d-e418850d25b6\" (UID: \"116e4234-31db-4d7f-b00d-e418850d25b6\") "
Nov 02 18:13:18.046192 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.046151    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/116e4234-31db-4d7f-b00d-e418850d25b6-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116e4234-31db-4d7f-b00d-e418850d25b6-maas-api-xks-ca\") pod \"116e4234-31db-4d7f-b00d-e418850d25b6\" (UID: \"116e4234-31db-4d7f-b00d-e418850d25b6\") "
Nov 02 18:13:18.046463 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.046440    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/116e4234-31db-4d7f-b00d-e418850d25b6-maas-api-openshift-ca" pod "116e4234-31db-4d7f-b00d-e418850d25b6" (UID: "116e4234-31db-4d7f-b00d-e418850d25b6"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:18.046733 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.046708    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/116e4234-31db-4d7f-b00d-e418850d25b6-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:18.048317 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.048294    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/116e4234-31db-4d7f-b00d-e418850d25b6-kube-api-access-lhgnj" pod "116e4234-31db-4d7f-b00d-e418850d25b6" (UID: "116e4234-31db-4d7f-b00d-e418850d25b6"). InnerVolumeSpecName "kube-api-access-lhgnj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:18.048393 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.048354    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/116e4234-31db-4d7f-b00d-e418850d25b6-maas-api-xks-ca" pod "116e4234-31db-4d7f-b00d-e418850d25b6" (UID: "116e4234-31db-4d7f-b00d-e418850d25b6"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:18.147897 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.147787    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/116e4234-31db-4d7f-b00d-e418850d25b6-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:18.147897 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.147849    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lhgnj\" (UniqueName: \"kubernetes.io/projected/116e4234-31db-4d7f-b00d-e418850d25b6-kube-api-access-lhgnj\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:18.483573 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.483540    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:18.641230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.641198    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:18.705426 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.705391    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"]
Nov 02 18:13:18.705950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.705931    4456 cpu_manager.go:414] "containerMap: removing container" podUID="be3edfab-727f-442d-8fe3-176358694c84" containerName="revoke-keys"
Nov 02 18:13:18.705950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.705951    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="be3edfab-727f-442d-8fe3-176358694c84" containerName="revoke-keys"
Nov 02 18:13:18.706110 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.706008    4456 cpu_manager.go:414] "containerMap: removing container" podUID="116e4234-31db-4d7f-b00d-e418850d25b6" containerName="revoke-keys"
Nov 02 18:13:18.706110 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.706017    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="116e4234-31db-4d7f-b00d-e418850d25b6" containerName="revoke-keys"
Nov 02 18:13:18.706236 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.706221    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="116e4234-31db-4d7f-b00d-e418850d25b6" containerName="revoke-keys"
Nov 02 18:13:18.706288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.706248    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="be3edfab-727f-442d-8fe3-176358694c84" containerName="revoke-keys"
Nov 02 18:13:18.709778 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.709756    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"
Nov 02 18:13:18.729716 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.727130    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"]
Nov 02 18:13:18.731684 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.731661    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"
Nov 02 18:13:18.740440 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.740387    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"]
Nov 02 18:13:18.751407 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.751380    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"]
Nov 02 18:13:18.754946 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.754921    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf7cfcae-bnbxx\" (UID: \"cc0bc138-76e3-41ca-83ac-c32c5a6770ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"
Nov 02 18:13:18.755069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.755000    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-w54kv\" (UniqueName: \"kubernetes.io/projected/ac13cb59-6208-4268-818f-205e6105789b-kube-api-access-w54kv\") pod \"maas-api-revoke-sub-be9a29be-6qzx5\" (UID: \"ac13cb59-6208-4268-818f-205e6105789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"
Nov 02 18:13:18.755069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.755049    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-cszsp\" (UniqueName: \"kubernetes.io/projected/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-kube-api-access-cszsp\") pod \"maas-api-revoke-sub-bf7cfcae-bnbxx\" (UID: \"cc0bc138-76e3-41ca-83ac-c32c5a6770ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"
Nov 02 18:13:18.755192 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.755072    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac13cb59-6208-4268-818f-205e6105789b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be9a29be-6qzx5\" (UID: \"ac13cb59-6208-4268-818f-205e6105789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"
Nov 02 18:13:18.755192 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.755108    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac13cb59-6208-4268-818f-205e6105789b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be9a29be-6qzx5\" (UID: \"ac13cb59-6208-4268-818f-205e6105789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"
Nov 02 18:13:18.755295 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.755260    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf7cfcae-bnbxx\" (UID: \"cc0bc138-76e3-41ca-83ac-c32c5a6770ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"
Nov 02 18:13:18.856745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.856655    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-cszsp\" (UniqueName: \"kubernetes.io/projected/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-kube-api-access-cszsp\") pod \"maas-api-revoke-sub-bf7cfcae-bnbxx\" (UID: \"cc0bc138-76e3-41ca-83ac-c32c5a6770ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"
Nov 02 18:13:18.856745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.856687    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac13cb59-6208-4268-818f-205e6105789b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be9a29be-6qzx5\" (UID: \"ac13cb59-6208-4268-818f-205e6105789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"
Nov 02 18:13:18.856745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.856714    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac13cb59-6208-4268-818f-205e6105789b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be9a29be-6qzx5\" (UID: \"ac13cb59-6208-4268-818f-205e6105789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"
Nov 02 18:13:18.856904 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.856781    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf7cfcae-bnbxx\" (UID: \"cc0bc138-76e3-41ca-83ac-c32c5a6770ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"
Nov 02 18:13:18.856960 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.856921    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf7cfcae-bnbxx\" (UID: \"cc0bc138-76e3-41ca-83ac-c32c5a6770ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"
Nov 02 18:13:18.857096 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.856999    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-w54kv\" (UniqueName: \"kubernetes.io/projected/ac13cb59-6208-4268-818f-205e6105789b-kube-api-access-w54kv\") pod \"maas-api-revoke-sub-be9a29be-6qzx5\" (UID: \"ac13cb59-6208-4268-818f-205e6105789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"
Nov 02 18:13:18.857477 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.857455    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac13cb59-6208-4268-818f-205e6105789b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-be9a29be-6qzx5\" (UID: \"ac13cb59-6208-4268-818f-205e6105789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"
Nov 02 18:13:18.857883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.857755    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-bf7cfcae-bnbxx\" (UID: \"cc0bc138-76e3-41ca-83ac-c32c5a6770ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"
Nov 02 18:13:18.859571 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.859547    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac13cb59-6208-4268-818f-205e6105789b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-be9a29be-6qzx5\" (UID: \"ac13cb59-6208-4268-818f-205e6105789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"
Nov 02 18:13:18.859641 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.859594    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-maas-api-xks-ca\") pod \"maas-api-revoke-sub-bf7cfcae-bnbxx\" (UID: \"cc0bc138-76e3-41ca-83ac-c32c5a6770ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"
Nov 02 18:13:18.881264 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.881243    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-w54kv\" (UniqueName: \"kubernetes.io/projected/ac13cb59-6208-4268-818f-205e6105789b-kube-api-access-w54kv\") pod \"maas-api-revoke-sub-be9a29be-6qzx5\" (UID: \"ac13cb59-6208-4268-818f-205e6105789b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"
Nov 02 18:13:18.881633 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.881614    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-cszsp\" (UniqueName: \"kubernetes.io/projected/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-kube-api-access-cszsp\") pod \"maas-api-revoke-sub-bf7cfcae-bnbxx\" (UID: \"cc0bc138-76e3-41ca-83ac-c32c5a6770ee\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"
Nov 02 18:13:18.894392 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.894368    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"
Nov 02 18:13:18.894485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.894383    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p" event={"ID":"116e4234-31db-4d7f-b00d-e418850d25b6","Type":"ContainerDied","Data":"72b2ada97787a5119c8c575f8954d6ad3fc50361b60af8ceeea039d72d5192eb"}
Nov 02 18:13:18.894485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:18.894433    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="72b2ada97787a5119c8c575f8954d6ad3fc50361b60af8ceeea039d72d5192eb"
Nov 02 18:13:19.030851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.029718    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"
Nov 02 18:13:19.050685 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.050331    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"
Nov 02 18:13:19.136458 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.136428    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 18:13:19.161215 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.161180    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-token\" (UniqueName: \"kubernetes.io/projected/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-token\") pod \"2dc48441-ccc0-47fc-abe7-7c327559eba4\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") "
Nov 02 18:13:19.161341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.161255    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-credential-socket\") pod \"2dc48441-ccc0-47fc-abe7-7c327559eba4\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") "
Nov 02 18:13:19.161341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.161330    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-data\") pod \"2dc48441-ccc0-47fc-abe7-7c327559eba4\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") "
Nov 02 18:13:19.161459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.161396    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/2dc48441-ccc0-47fc-abe7-7c327559eba4-kube-api-access-d6rw4\" (UniqueName: \"kubernetes.io/projected/2dc48441-ccc0-47fc-abe7-7c327559eba4-kube-api-access-d6rw4\") pod \"2dc48441-ccc0-47fc-abe7-7c327559eba4\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") "
Nov 02 18:13:19.161459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.161420    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-envoy\") pod \"2dc48441-ccc0-47fc-abe7-7c327559eba4\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") "
Nov 02 18:13:19.161569 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.161456    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/2dc48441-ccc0-47fc-abe7-7c327559eba4-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc48441-ccc0-47fc-abe7-7c327559eba4-istiod-ca-cert\") pod \"2dc48441-ccc0-47fc-abe7-7c327559eba4\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") "
Nov 02 18:13:19.161569 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.161549    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-workload-certs\") pod \"2dc48441-ccc0-47fc-abe7-7c327559eba4\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") "
Nov 02 18:13:19.161669 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.161600    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-workload-socket\") pod \"2dc48441-ccc0-47fc-abe7-7c327559eba4\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") "
Nov 02 18:13:19.161669 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.161621    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-podinfo\") pod \"2dc48441-ccc0-47fc-abe7-7c327559eba4\" (UID: \"2dc48441-ccc0-47fc-abe7-7c327559eba4\") "
Nov 02 18:13:19.162989 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.162665    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-workload-certs" pod "2dc48441-ccc0-47fc-abe7-7c327559eba4" (UID: "2dc48441-ccc0-47fc-abe7-7c327559eba4"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:19.162989 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.162918    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-credential-socket" pod "2dc48441-ccc0-47fc-abe7-7c327559eba4" (UID: "2dc48441-ccc0-47fc-abe7-7c327559eba4"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:19.162989 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.162968    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-workload-socket" pod "2dc48441-ccc0-47fc-abe7-7c327559eba4" (UID: "2dc48441-ccc0-47fc-abe7-7c327559eba4"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:19.163232 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.163119    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-data" pod "2dc48441-ccc0-47fc-abe7-7c327559eba4" (UID: "2dc48441-ccc0-47fc-abe7-7c327559eba4"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:19.163232 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.163155    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/2dc48441-ccc0-47fc-abe7-7c327559eba4-istiod-ca-cert" pod "2dc48441-ccc0-47fc-abe7-7c327559eba4" (UID: "2dc48441-ccc0-47fc-abe7-7c327559eba4"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:19.165395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.165373    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-podinfo" pod "2dc48441-ccc0-47fc-abe7-7c327559eba4" (UID: "2dc48441-ccc0-47fc-abe7-7c327559eba4"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:13:19.165500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.165425    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc48441-ccc0-47fc-abe7-7c327559eba4-kube-api-access-d6rw4" pod "2dc48441-ccc0-47fc-abe7-7c327559eba4" (UID: "2dc48441-ccc0-47fc-abe7-7c327559eba4"). InnerVolumeSpecName "kube-api-access-d6rw4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:19.166568 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.166544    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-token" pod "2dc48441-ccc0-47fc-abe7-7c327559eba4" (UID: "2dc48441-ccc0-47fc-abe7-7c327559eba4"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:19.166656 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.166598    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-envoy" pod "2dc48441-ccc0-47fc-abe7-7c327559eba4" (UID: "2dc48441-ccc0-47fc-abe7-7c327559eba4"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:19.169241 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.169169    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:19.262635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.262604    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:19.262635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.262635    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:19.262879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.262648    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:19.262879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.262664    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d6rw4\" (UniqueName: \"kubernetes.io/projected/2dc48441-ccc0-47fc-abe7-7c327559eba4-kube-api-access-d6rw4\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:19.262879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.262678    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:19.262879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.262689    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/2dc48441-ccc0-47fc-abe7-7c327559eba4-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:19.262879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.262699    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:19.262879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.262709    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/2dc48441-ccc0-47fc-abe7-7c327559eba4-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:19.262879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.262720    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/2dc48441-ccc0-47fc-abe7-7c327559eba4-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:19.322751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.322441    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"]
Nov 02 18:13:19.343248 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.343212    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"]
Nov 02 18:13:19.364801 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:13:19.364774    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcc0bc138_76e3_41ca_83ac_c32c5a6770ee.slice/crio-3e41231dfa7a340d28a0954050cd95f38f378edfa0863456ae23c1c74816b0c8 WatchSource:0}: Error finding container 3e41231dfa7a340d28a0954050cd95f38f378edfa0863456ae23c1c74816b0c8: Status 404 returned error can't find the container with id 3e41231dfa7a340d28a0954050cd95f38f378edfa0863456ae23c1c74816b0c8
Nov 02 18:13:19.365894 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.365849    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"]
Nov 02 18:13:19.396458 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.396432    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"]
Nov 02 18:13:19.400467 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:13:19.400437    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podac13cb59_6208_4268_818f_205e6105789b.slice/crio-3d58f5732f7244648eaa71f44a95fce702b3b84dbd94bb3b90c12978c2553294 WatchSource:0}: Error finding container 3d58f5732f7244648eaa71f44a95fce702b3b84dbd94bb3b90c12978c2553294: Status 404 returned error can't find the container with id 3d58f5732f7244648eaa71f44a95fce702b3b84dbd94bb3b90c12978c2553294
Nov 02 18:13:19.512824 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.512791    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:19.906249 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.906158    4456 generic.go:356] "Generic (PLEG): container finished" podID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerID="73442968debcfc6a6635d7a8712e48213ab8ebc83bdaf93b458f2f9b8b758f89" exitCode=0
Nov 02 18:13:19.906414 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.906252    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" event={"ID":"2dc48441-ccc0-47fc-abe7-7c327559eba4","Type":"ContainerDied","Data":"73442968debcfc6a6635d7a8712e48213ab8ebc83bdaf93b458f2f9b8b758f89"}
Nov 02 18:13:19.909859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.909780    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j" event={"ID":"2dc48441-ccc0-47fc-abe7-7c327559eba4","Type":"ContainerDied","Data":"0b3d9d3727e971980971b177f1eac5f801f346dfcc99842d0439d668b81f28ad"}
Nov 02 18:13:19.909859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.909854    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5" event={"ID":"ac13cb59-6208-4268-818f-205e6105789b","Type":"ContainerStarted","Data":"5aba2d1b65f569de765561cf90205073daed5f1814dd4b9383bdc6e67ac6aa57"}
Nov 02 18:13:19.910070 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.909883    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5" event={"ID":"ac13cb59-6208-4268-818f-205e6105789b","Type":"ContainerStarted","Data":"3d58f5732f7244648eaa71f44a95fce702b3b84dbd94bb3b90c12978c2553294"}
Nov 02 18:13:19.910070 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.909916    4456 scope.go:122] "RemoveContainer" containerID="73442968debcfc6a6635d7a8712e48213ab8ebc83bdaf93b458f2f9b8b758f89"
Nov 02 18:13:19.910189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.910106    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"
Nov 02 18:13:19.912949 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.912920    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx" event={"ID":"cc0bc138-76e3-41ca-83ac-c32c5a6770ee","Type":"ContainerStarted","Data":"8ecf27952d156cd1e1347837f3c19cb86b6ce3e4619982d42429340e53ef7aed"}
Nov 02 18:13:19.913061 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.912973    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx" event={"ID":"cc0bc138-76e3-41ca-83ac-c32c5a6770ee","Type":"ContainerStarted","Data":"3e41231dfa7a340d28a0954050cd95f38f378edfa0863456ae23c1c74816b0c8"}
Nov 02 18:13:19.922551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.922530    4456 scope.go:122] "RemoveContainer" containerID="1a22dd302b6925abeb1cc285dfdf5df0473f58e4b460c5dab034c0a14fa18275"
Nov 02 18:13:19.933327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.933305    4456 scope.go:122] "RemoveContainer" containerID="73442968debcfc6a6635d7a8712e48213ab8ebc83bdaf93b458f2f9b8b758f89"
Nov 02 18:13:19.933634 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:19.933607    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"73442968debcfc6a6635d7a8712e48213ab8ebc83bdaf93b458f2f9b8b758f89\": container with ID starting with 73442968debcfc6a6635d7a8712e48213ab8ebc83bdaf93b458f2f9b8b758f89 not found: ID does not exist" containerID="73442968debcfc6a6635d7a8712e48213ab8ebc83bdaf93b458f2f9b8b758f89"
Nov 02 18:13:19.933683 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.933649    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"73442968debcfc6a6635d7a8712e48213ab8ebc83bdaf93b458f2f9b8b758f89"} err="failed to get container status \"73442968debcfc6a6635d7a8712e48213ab8ebc83bdaf93b458f2f9b8b758f89\": rpc error: code = NotFound desc = could not find container \"73442968debcfc6a6635d7a8712e48213ab8ebc83bdaf93b458f2f9b8b758f89\": container with ID starting with 73442968debcfc6a6635d7a8712e48213ab8ebc83bdaf93b458f2f9b8b758f89 not found: ID does not exist"
Nov 02 18:13:19.933683 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.933668    4456 scope.go:122] "RemoveContainer" containerID="1a22dd302b6925abeb1cc285dfdf5df0473f58e4b460c5dab034c0a14fa18275"
Nov 02 18:13:19.933990 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:19.933964    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a22dd302b6925abeb1cc285dfdf5df0473f58e4b460c5dab034c0a14fa18275\": container with ID starting with 1a22dd302b6925abeb1cc285dfdf5df0473f58e4b460c5dab034c0a14fa18275 not found: ID does not exist" containerID="1a22dd302b6925abeb1cc285dfdf5df0473f58e4b460c5dab034c0a14fa18275"
Nov 02 18:13:19.934070 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.934002    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a22dd302b6925abeb1cc285dfdf5df0473f58e4b460c5dab034c0a14fa18275"} err="failed to get container status \"1a22dd302b6925abeb1cc285dfdf5df0473f58e4b460c5dab034c0a14fa18275\": rpc error: code = NotFound desc = could not find container \"1a22dd302b6925abeb1cc285dfdf5df0473f58e4b460c5dab034c0a14fa18275\": container with ID starting with 1a22dd302b6925abeb1cc285dfdf5df0473f58e4b460c5dab034c0a14fa18275 not found: ID does not exist"
Nov 02 18:13:19.958808 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:19.958745    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5" podStartSLOduration=1.958727434 podStartE2EDuration="1.958727434s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:19.957213133 +0000 UTC m=+2616.448276293" watchObservedRunningTime="2026-11-02 18:13:19.958727434 +0000 UTC m=+2616.449790597"
Nov 02 18:13:20.015474 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.015442    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"]
Nov 02 18:13:20.073594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.073557    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-b-w5-c810ca-openshift-default-6d99bf666c-9q28j"]
Nov 02 18:13:20.128747 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.128679    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx" podStartSLOduration=2.128659397 podStartE2EDuration="2.128659397s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:18 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:20.125308729 +0000 UTC m=+2616.616371890" watchObservedRunningTime="2026-11-02 18:13:20.128659397 +0000 UTC m=+2616.619722562"
Nov 02 18:13:20.516196 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.516161    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:20.755659 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.755626    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:20.849730 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.849689    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"]
Nov 02 18:13:20.850293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.850269    4456 cpu_manager.go:414] "containerMap: removing container" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy"
Nov 02 18:13:20.850293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.850292    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy"
Nov 02 18:13:20.850523 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.850457    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy"
Nov 02 18:13:20.850523 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.850477    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy"
Nov 02 18:13:20.853921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.853890    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"
Nov 02 18:13:20.877179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.877151    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"]
Nov 02 18:13:20.884518 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.884490    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e153e5-bd9d-4a00-8c94-75f6a53af87c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd103858-cjlgg\" (UID: \"18e153e5-bd9d-4a00-8c94-75f6a53af87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"
Nov 02 18:13:20.885590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.884524    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e153e5-bd9d-4a00-8c94-75f6a53af87c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd103858-cjlgg\" (UID: \"18e153e5-bd9d-4a00-8c94-75f6a53af87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"
Nov 02 18:13:20.885590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.884662    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c7vm4\" (UniqueName: \"kubernetes.io/projected/18e153e5-bd9d-4a00-8c94-75f6a53af87c-kube-api-access-c7vm4\") pod \"maas-api-revoke-sub-cd103858-cjlgg\" (UID: \"18e153e5-bd9d-4a00-8c94-75f6a53af87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"
Nov 02 18:13:20.889185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.889164    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"]
Nov 02 18:13:20.889679 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.889662    4456 cpu_manager.go:414] "containerMap: removing container" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy"
Nov 02 18:13:20.889679 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.889680    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" containerName="istio-proxy"
Nov 02 18:13:20.893030 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.893014    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"
Nov 02 18:13:20.915256 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.915204    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"]
Nov 02 18:13:20.925927 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.925626    4456 generic.go:356] "Generic (PLEG): container finished" podID="ac13cb59-6208-4268-818f-205e6105789b" containerID="5aba2d1b65f569de765561cf90205073daed5f1814dd4b9383bdc6e67ac6aa57" exitCode=0
Nov 02 18:13:20.925927 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.925714    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5" event={"ID":"ac13cb59-6208-4268-818f-205e6105789b","Type":"ContainerDied","Data":"5aba2d1b65f569de765561cf90205073daed5f1814dd4b9383bdc6e67ac6aa57"}
Nov 02 18:13:20.927797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.927769    4456 generic.go:356] "Generic (PLEG): container finished" podID="cc0bc138-76e3-41ca-83ac-c32c5a6770ee" containerID="8ecf27952d156cd1e1347837f3c19cb86b6ce3e4619982d42429340e53ef7aed" exitCode=0
Nov 02 18:13:20.927935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.927877    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx" event={"ID":"cc0bc138-76e3-41ca-83ac-c32c5a6770ee","Type":"ContainerDied","Data":"8ecf27952d156cd1e1347837f3c19cb86b6ce3e4619982d42429340e53ef7aed"}
Nov 02 18:13:20.986267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.986232    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c7vm4\" (UniqueName: \"kubernetes.io/projected/18e153e5-bd9d-4a00-8c94-75f6a53af87c-kube-api-access-c7vm4\") pod \"maas-api-revoke-sub-cd103858-cjlgg\" (UID: \"18e153e5-bd9d-4a00-8c94-75f6a53af87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"
Nov 02 18:13:20.986441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.986299    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/887b756a-8d5e-4510-9f48-0223365a0543-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0b090ef-7b5wk\" (UID: \"887b756a-8d5e-4510-9f48-0223365a0543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"
Nov 02 18:13:20.986441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.986378    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e153e5-bd9d-4a00-8c94-75f6a53af87c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd103858-cjlgg\" (UID: \"18e153e5-bd9d-4a00-8c94-75f6a53af87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"
Nov 02 18:13:20.986516 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.986459    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e153e5-bd9d-4a00-8c94-75f6a53af87c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd103858-cjlgg\" (UID: \"18e153e5-bd9d-4a00-8c94-75f6a53af87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"
Nov 02 18:13:20.986578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.986555    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hfcqz\" (UniqueName: \"kubernetes.io/projected/887b756a-8d5e-4510-9f48-0223365a0543-kube-api-access-hfcqz\") pod \"maas-api-revoke-sub-e0b090ef-7b5wk\" (UID: \"887b756a-8d5e-4510-9f48-0223365a0543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"
Nov 02 18:13:20.986625 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.986597    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/887b756a-8d5e-4510-9f48-0223365a0543-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0b090ef-7b5wk\" (UID: \"887b756a-8d5e-4510-9f48-0223365a0543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"
Nov 02 18:13:20.987227 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.987204    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e153e5-bd9d-4a00-8c94-75f6a53af87c-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-cd103858-cjlgg\" (UID: \"18e153e5-bd9d-4a00-8c94-75f6a53af87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"
Nov 02 18:13:20.989240 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:20.989219    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e153e5-bd9d-4a00-8c94-75f6a53af87c-maas-api-xks-ca\") pod \"maas-api-revoke-sub-cd103858-cjlgg\" (UID: \"18e153e5-bd9d-4a00-8c94-75f6a53af87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"
Nov 02 18:13:21.012474 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.012401    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c7vm4\" (UniqueName: \"kubernetes.io/projected/18e153e5-bd9d-4a00-8c94-75f6a53af87c-kube-api-access-c7vm4\") pod \"maas-api-revoke-sub-cd103858-cjlgg\" (UID: \"18e153e5-bd9d-4a00-8c94-75f6a53af87c\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"
Nov 02 18:13:21.088331 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.088288    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/887b756a-8d5e-4510-9f48-0223365a0543-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0b090ef-7b5wk\" (UID: \"887b756a-8d5e-4510-9f48-0223365a0543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"
Nov 02 18:13:21.088785 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.088475    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hfcqz\" (UniqueName: \"kubernetes.io/projected/887b756a-8d5e-4510-9f48-0223365a0543-kube-api-access-hfcqz\") pod \"maas-api-revoke-sub-e0b090ef-7b5wk\" (UID: \"887b756a-8d5e-4510-9f48-0223365a0543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"
Nov 02 18:13:21.088785 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.088505    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/887b756a-8d5e-4510-9f48-0223365a0543-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0b090ef-7b5wk\" (UID: \"887b756a-8d5e-4510-9f48-0223365a0543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"
Nov 02 18:13:21.091886 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.089379    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/887b756a-8d5e-4510-9f48-0223365a0543-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e0b090ef-7b5wk\" (UID: \"887b756a-8d5e-4510-9f48-0223365a0543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"
Nov 02 18:13:21.094855 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.094829    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/887b756a-8d5e-4510-9f48-0223365a0543-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e0b090ef-7b5wk\" (UID: \"887b756a-8d5e-4510-9f48-0223365a0543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"
Nov 02 18:13:21.114292 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.114266    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hfcqz\" (UniqueName: \"kubernetes.io/projected/887b756a-8d5e-4510-9f48-0223365a0543-kube-api-access-hfcqz\") pod \"maas-api-revoke-sub-e0b090ef-7b5wk\" (UID: \"887b756a-8d5e-4510-9f48-0223365a0543\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"
Nov 02 18:13:21.167614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.167584    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"
Nov 02 18:13:21.213324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.213284    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"
Nov 02 18:13:21.277899 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.275223    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:21.472490 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.472461    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"]
Nov 02 18:13:21.473360 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:13:21.473337    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod18e153e5_bd9d_4a00_8c94_75f6a53af87c.slice/crio-f423e62e8cb5cb7f3f642c7de3e8beb6372dd1dd90fa8fe4965dcab170f11b2b WatchSource:0}: Error finding container f423e62e8cb5cb7f3f642c7de3e8beb6372dd1dd90fa8fe4965dcab170f11b2b: Status 404 returned error can't find the container with id f423e62e8cb5cb7f3f642c7de3e8beb6372dd1dd90fa8fe4965dcab170f11b2b
Nov 02 18:13:21.545118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.545090    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"]
Nov 02 18:13:21.545631 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:13:21.545606    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod887b756a_8d5e_4510_9f48_0223365a0543.slice/crio-cafdd1f6190bfd30388f2ed9bac6dfa7b54e440bce6f3cc5576025b824891a41 WatchSource:0}: Error finding container cafdd1f6190bfd30388f2ed9bac6dfa7b54e440bce6f3cc5576025b824891a41: Status 404 returned error can't find the container with id cafdd1f6190bfd30388f2ed9bac6dfa7b54e440bce6f3cc5576025b824891a41
Nov 02 18:13:21.612840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.612788    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2dc48441-ccc0-47fc-abe7-7c327559eba4" path="/var/lib/kubelet/pods/2dc48441-ccc0-47fc-abe7-7c327559eba4/volumes"
Nov 02 18:13:21.935668 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.935634    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg" event={"ID":"18e153e5-bd9d-4a00-8c94-75f6a53af87c","Type":"ContainerStarted","Data":"cb76a2fc0415f3de7c9aff049e81d8ba17af8974170662e2fee6da448986a4be"}
Nov 02 18:13:21.935843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.935704    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg" event={"ID":"18e153e5-bd9d-4a00-8c94-75f6a53af87c","Type":"ContainerStarted","Data":"f423e62e8cb5cb7f3f642c7de3e8beb6372dd1dd90fa8fe4965dcab170f11b2b"}
Nov 02 18:13:21.937375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.937336    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk" event={"ID":"887b756a-8d5e-4510-9f48-0223365a0543","Type":"ContainerStarted","Data":"8568021d7fe7c61c983b4ad67a269994f91f4881e20e8f9ee6516ab6dd0bf82a"}
Nov 02 18:13:21.937481 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.937397    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk" event={"ID":"887b756a-8d5e-4510-9f48-0223365a0543","Type":"ContainerStarted","Data":"cafdd1f6190bfd30388f2ed9bac6dfa7b54e440bce6f3cc5576025b824891a41"}
Nov 02 18:13:21.981724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:21.981506    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg" podStartSLOduration=1.981491951 podStartE2EDuration="1.981491951s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:21.977859439 +0000 UTC m=+2618.468922613" watchObservedRunningTime="2026-11-02 18:13:21.981491951 +0000 UTC m=+2618.472555113"
Nov 02 18:13:22.146283 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.146257    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"
Nov 02 18:13:22.174904 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.174880    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"
Nov 02 18:13:22.192730 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.192677    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk" podStartSLOduration=2.192662755 podStartE2EDuration="2.192662755s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:13:20 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:13:22.023602029 +0000 UTC m=+2618.514665194" watchObservedRunningTime="2026-11-02 18:13:22.192662755 +0000 UTC m=+2618.683725916"
Nov 02 18:13:22.202637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.202607    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-maas-api-openshift-ca\") pod \"cc0bc138-76e3-41ca-83ac-c32c5a6770ee\" (UID: \"cc0bc138-76e3-41ca-83ac-c32c5a6770ee\") "
Nov 02 18:13:22.202793 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.202778    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ac13cb59-6208-4268-818f-205e6105789b-kube-api-access-w54kv\" (UniqueName: \"kubernetes.io/projected/ac13cb59-6208-4268-818f-205e6105789b-kube-api-access-w54kv\") pod \"ac13cb59-6208-4268-818f-205e6105789b\" (UID: \"ac13cb59-6208-4268-818f-205e6105789b\") "
Nov 02 18:13:22.202896 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.202857    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-kube-api-access-cszsp\" (UniqueName: \"kubernetes.io/projected/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-kube-api-access-cszsp\") pod \"cc0bc138-76e3-41ca-83ac-c32c5a6770ee\" (UID: \"cc0bc138-76e3-41ca-83ac-c32c5a6770ee\") "
Nov 02 18:13:22.202954 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.202894    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ac13cb59-6208-4268-818f-205e6105789b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac13cb59-6208-4268-818f-205e6105789b-maas-api-openshift-ca\") pod \"ac13cb59-6208-4268-818f-205e6105789b\" (UID: \"ac13cb59-6208-4268-818f-205e6105789b\") "
Nov 02 18:13:22.202954 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.202926    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-maas-api-xks-ca\") pod \"cc0bc138-76e3-41ca-83ac-c32c5a6770ee\" (UID: \"cc0bc138-76e3-41ca-83ac-c32c5a6770ee\") "
Nov 02 18:13:22.203050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.202956    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/ac13cb59-6208-4268-818f-205e6105789b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac13cb59-6208-4268-818f-205e6105789b-maas-api-xks-ca\") pod \"ac13cb59-6208-4268-818f-205e6105789b\" (UID: \"ac13cb59-6208-4268-818f-205e6105789b\") "
Nov 02 18:13:22.203131 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.203081    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-maas-api-openshift-ca" pod "cc0bc138-76e3-41ca-83ac-c32c5a6770ee" (UID: "cc0bc138-76e3-41ca-83ac-c32c5a6770ee"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:22.203527 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.203505    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:22.204140 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.204116    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ac13cb59-6208-4268-818f-205e6105789b-maas-api-openshift-ca" pod "ac13cb59-6208-4268-818f-205e6105789b" (UID: "ac13cb59-6208-4268-818f-205e6105789b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:22.206610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.206581    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ac13cb59-6208-4268-818f-205e6105789b-kube-api-access-w54kv" pod "ac13cb59-6208-4268-818f-205e6105789b" (UID: "ac13cb59-6208-4268-818f-205e6105789b"). InnerVolumeSpecName "kube-api-access-w54kv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:22.206706 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.206644    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/ac13cb59-6208-4268-818f-205e6105789b-maas-api-xks-ca" pod "ac13cb59-6208-4268-818f-205e6105789b" (UID: "ac13cb59-6208-4268-818f-205e6105789b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:22.206945 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.206927    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-maas-api-xks-ca" pod "cc0bc138-76e3-41ca-83ac-c32c5a6770ee" (UID: "cc0bc138-76e3-41ca-83ac-c32c5a6770ee"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:22.207987 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.207956    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-kube-api-access-cszsp" pod "cc0bc138-76e3-41ca-83ac-c32c5a6770ee" (UID: "cc0bc138-76e3-41ca-83ac-c32c5a6770ee"). InnerVolumeSpecName "kube-api-access-cszsp". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:22.305178 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.305139    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cszsp\" (UniqueName: \"kubernetes.io/projected/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-kube-api-access-cszsp\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:22.305178 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.305176    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/ac13cb59-6208-4268-818f-205e6105789b-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:22.305414 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.305192    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/cc0bc138-76e3-41ca-83ac-c32c5a6770ee-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:22.305414 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.305206    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/ac13cb59-6208-4268-818f-205e6105789b-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:22.305414 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.305219    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-w54kv\" (UniqueName: \"kubernetes.io/projected/ac13cb59-6208-4268-818f-205e6105789b-kube-api-access-w54kv\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:22.946364 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.946321    4456 generic.go:356] "Generic (PLEG): container finished" podID="18e153e5-bd9d-4a00-8c94-75f6a53af87c" containerID="cb76a2fc0415f3de7c9aff049e81d8ba17af8974170662e2fee6da448986a4be" exitCode=0
Nov 02 18:13:22.946548 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.946406    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg" event={"ID":"18e153e5-bd9d-4a00-8c94-75f6a53af87c","Type":"ContainerDied","Data":"cb76a2fc0415f3de7c9aff049e81d8ba17af8974170662e2fee6da448986a4be"}
Nov 02 18:13:22.948169 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.948141    4456 generic.go:356] "Generic (PLEG): container finished" podID="887b756a-8d5e-4510-9f48-0223365a0543" containerID="8568021d7fe7c61c983b4ad67a269994f91f4881e20e8f9ee6516ab6dd0bf82a" exitCode=0
Nov 02 18:13:22.948281 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.948247    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk" event={"ID":"887b756a-8d5e-4510-9f48-0223365a0543","Type":"ContainerDied","Data":"8568021d7fe7c61c983b4ad67a269994f91f4881e20e8f9ee6516ab6dd0bf82a"}
Nov 02 18:13:22.949744 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.949722    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"
Nov 02 18:13:22.949898 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.949776    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5" event={"ID":"ac13cb59-6208-4268-818f-205e6105789b","Type":"ContainerDied","Data":"3d58f5732f7244648eaa71f44a95fce702b3b84dbd94bb3b90c12978c2553294"}
Nov 02 18:13:22.949898 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.949840    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3d58f5732f7244648eaa71f44a95fce702b3b84dbd94bb3b90c12978c2553294"
Nov 02 18:13:22.951485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.951455    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx" event={"ID":"cc0bc138-76e3-41ca-83ac-c32c5a6770ee","Type":"ContainerDied","Data":"3e41231dfa7a340d28a0954050cd95f38f378edfa0863456ae23c1c74816b0c8"}
Nov 02 18:13:22.951593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.951502    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3e41231dfa7a340d28a0954050cd95f38f378edfa0863456ae23c1c74816b0c8"
Nov 02 18:13:22.951593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.951464    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"
Nov 02 18:13:22.953911 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.953892    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"]
Nov 02 18:13:22.989401 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:22.989365    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"]
Nov 02 18:13:23.144375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:23.144334    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:23.296163 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:23.296052    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:24.140832 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.140796    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"
Nov 02 18:13:24.144037 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.144020    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"
Nov 02 18:13:24.229848 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.229692    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/887b756a-8d5e-4510-9f48-0223365a0543-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/887b756a-8d5e-4510-9f48-0223365a0543-maas-api-openshift-ca\") pod \"887b756a-8d5e-4510-9f48-0223365a0543\" (UID: \"887b756a-8d5e-4510-9f48-0223365a0543\") "
Nov 02 18:13:24.231803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.229848    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/18e153e5-bd9d-4a00-8c94-75f6a53af87c-kube-api-access-c7vm4\" (UniqueName: \"kubernetes.io/projected/18e153e5-bd9d-4a00-8c94-75f6a53af87c-kube-api-access-c7vm4\") pod \"18e153e5-bd9d-4a00-8c94-75f6a53af87c\" (UID: \"18e153e5-bd9d-4a00-8c94-75f6a53af87c\") "
Nov 02 18:13:24.231803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.230034    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/887b756a-8d5e-4510-9f48-0223365a0543-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/887b756a-8d5e-4510-9f48-0223365a0543-maas-api-xks-ca\") pod \"887b756a-8d5e-4510-9f48-0223365a0543\" (UID: \"887b756a-8d5e-4510-9f48-0223365a0543\") "
Nov 02 18:13:24.231803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.230064    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/18e153e5-bd9d-4a00-8c94-75f6a53af87c-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e153e5-bd9d-4a00-8c94-75f6a53af87c-maas-api-xks-ca\") pod \"18e153e5-bd9d-4a00-8c94-75f6a53af87c\" (UID: \"18e153e5-bd9d-4a00-8c94-75f6a53af87c\") "
Nov 02 18:13:24.231803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.230099    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/18e153e5-bd9d-4a00-8c94-75f6a53af87c-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e153e5-bd9d-4a00-8c94-75f6a53af87c-maas-api-openshift-ca\") pod \"18e153e5-bd9d-4a00-8c94-75f6a53af87c\" (UID: \"18e153e5-bd9d-4a00-8c94-75f6a53af87c\") "
Nov 02 18:13:24.231803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.230136    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/887b756a-8d5e-4510-9f48-0223365a0543-kube-api-access-hfcqz\" (UniqueName: \"kubernetes.io/projected/887b756a-8d5e-4510-9f48-0223365a0543-kube-api-access-hfcqz\") pod \"887b756a-8d5e-4510-9f48-0223365a0543\" (UID: \"887b756a-8d5e-4510-9f48-0223365a0543\") "
Nov 02 18:13:24.232112 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.231451    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/18e153e5-bd9d-4a00-8c94-75f6a53af87c-maas-api-openshift-ca" pod "18e153e5-bd9d-4a00-8c94-75f6a53af87c" (UID: "18e153e5-bd9d-4a00-8c94-75f6a53af87c"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:24.232167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.232150    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/887b756a-8d5e-4510-9f48-0223365a0543-maas-api-openshift-ca" pod "887b756a-8d5e-4510-9f48-0223365a0543" (UID: "887b756a-8d5e-4510-9f48-0223365a0543"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:24.233346 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.233317    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/18e153e5-bd9d-4a00-8c94-75f6a53af87c-maas-api-xks-ca" pod "18e153e5-bd9d-4a00-8c94-75f6a53af87c" (UID: "18e153e5-bd9d-4a00-8c94-75f6a53af87c"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:24.233583 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.233561    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/887b756a-8d5e-4510-9f48-0223365a0543-kube-api-access-hfcqz" pod "887b756a-8d5e-4510-9f48-0223365a0543" (UID: "887b756a-8d5e-4510-9f48-0223365a0543"). InnerVolumeSpecName "kube-api-access-hfcqz". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:24.234022 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.233995    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/18e153e5-bd9d-4a00-8c94-75f6a53af87c-kube-api-access-c7vm4" pod "18e153e5-bd9d-4a00-8c94-75f6a53af87c" (UID: "18e153e5-bd9d-4a00-8c94-75f6a53af87c"). InnerVolumeSpecName "kube-api-access-c7vm4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:24.234307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.234278    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/887b756a-8d5e-4510-9f48-0223365a0543-maas-api-xks-ca" pod "887b756a-8d5e-4510-9f48-0223365a0543" (UID: "887b756a-8d5e-4510-9f48-0223365a0543"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:24.331455 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.331417    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c7vm4\" (UniqueName: \"kubernetes.io/projected/18e153e5-bd9d-4a00-8c94-75f6a53af87c-kube-api-access-c7vm4\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:24.331455 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.331453    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/887b756a-8d5e-4510-9f48-0223365a0543-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:24.331910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.331467    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/18e153e5-bd9d-4a00-8c94-75f6a53af87c-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:24.331910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.331483    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/18e153e5-bd9d-4a00-8c94-75f6a53af87c-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:24.331910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.331496    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-hfcqz\" (UniqueName: \"kubernetes.io/projected/887b756a-8d5e-4510-9f48-0223365a0543-kube-api-access-hfcqz\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:24.331910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.331510    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/887b756a-8d5e-4510-9f48-0223365a0543-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:24.851378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.851339    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:24.965094 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.965064    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"
Nov 02 18:13:24.965094 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.965085    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg" event={"ID":"18e153e5-bd9d-4a00-8c94-75f6a53af87c","Type":"ContainerDied","Data":"f423e62e8cb5cb7f3f642c7de3e8beb6372dd1dd90fa8fe4965dcab170f11b2b"}
Nov 02 18:13:24.965305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.965141    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f423e62e8cb5cb7f3f642c7de3e8beb6372dd1dd90fa8fe4965dcab170f11b2b"
Nov 02 18:13:24.966550 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.966528    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"
Nov 02 18:13:24.966550 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.966536    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk" event={"ID":"887b756a-8d5e-4510-9f48-0223365a0543","Type":"ContainerDied","Data":"cafdd1f6190bfd30388f2ed9bac6dfa7b54e440bce6f3cc5576025b824891a41"}
Nov 02 18:13:24.966672 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:24.966573    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="cafdd1f6190bfd30388f2ed9bac6dfa7b54e440bce6f3cc5576025b824891a41"
Nov 02 18:13:25.199444 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:25.199409    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:25.341785 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:25.341754    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:26.739762 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:26.738232    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:27.954958 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:27.954922    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" podUID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" containerName="payload-processing" containerID="cri-o://7c1de68e597eb58fefb3cd933a0ac134632987153b7d92161a06b331b295e194" gracePeriod=25
Nov 02 18:13:27.989743 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:27.989703    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j" podUID="e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" containerName="payload-pre-processing" containerID="cri-o://09f75adb956ba8d1b347af752e4d6d17af18d67cc64d9c62d56a97f64f5dde8e" gracePeriod=25
Nov 02 18:13:28.278411 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.278387    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 18:13:28.289948 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.289924    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"
Nov 02 18:13:28.378944 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.378913    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e5ae7b61-cd5f-4a38-b038-28aaa6991eb7-kube-api-access-wzv9g\" (UniqueName: \"kubernetes.io/projected/e5ae7b61-cd5f-4a38-b038-28aaa6991eb7-kube-api-access-wzv9g\") pod \"e5ae7b61-cd5f-4a38-b038-28aaa6991eb7\" (UID: \"e5ae7b61-cd5f-4a38-b038-28aaa6991eb7\") "
Nov 02 18:13:28.379125 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.379029    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bca2dabc-94e2-4875-ba3f-fa9332afe8dc-kube-api-access-8tqd5\" (UniqueName: \"kubernetes.io/projected/bca2dabc-94e2-4875-ba3f-fa9332afe8dc-kube-api-access-8tqd5\") pod \"bca2dabc-94e2-4875-ba3f-fa9332afe8dc\" (UID: \"bca2dabc-94e2-4875-ba3f-fa9332afe8dc\") "
Nov 02 18:13:28.379125 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.379060    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e5ae7b61-cd5f-4a38-b038-28aaa6991eb7-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ae7b61-cd5f-4a38-b038-28aaa6991eb7-plugins-config-volume\") pod \"e5ae7b61-cd5f-4a38-b038-28aaa6991eb7\" (UID: \"e5ae7b61-cd5f-4a38-b038-28aaa6991eb7\") "
Nov 02 18:13:28.379125 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.379107    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/bca2dabc-94e2-4875-ba3f-fa9332afe8dc-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca2dabc-94e2-4875-ba3f-fa9332afe8dc-plugins-config-volume\") pod \"bca2dabc-94e2-4875-ba3f-fa9332afe8dc\" (UID: \"bca2dabc-94e2-4875-ba3f-fa9332afe8dc\") "
Nov 02 18:13:28.379538 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.379515    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e5ae7b61-cd5f-4a38-b038-28aaa6991eb7-plugins-config-volume" pod "e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" (UID: "e5ae7b61-cd5f-4a38-b038-28aaa6991eb7"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:28.379640 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.379608    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/bca2dabc-94e2-4875-ba3f-fa9332afe8dc-plugins-config-volume" pod "bca2dabc-94e2-4875-ba3f-fa9332afe8dc" (UID: "bca2dabc-94e2-4875-ba3f-fa9332afe8dc"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:28.381348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.381319    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bca2dabc-94e2-4875-ba3f-fa9332afe8dc-kube-api-access-8tqd5" pod "bca2dabc-94e2-4875-ba3f-fa9332afe8dc" (UID: "bca2dabc-94e2-4875-ba3f-fa9332afe8dc"). InnerVolumeSpecName "kube-api-access-8tqd5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:28.381348 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.381336    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e5ae7b61-cd5f-4a38-b038-28aaa6991eb7-kube-api-access-wzv9g" pod "e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" (UID: "e5ae7b61-cd5f-4a38-b038-28aaa6991eb7"). InnerVolumeSpecName "kube-api-access-wzv9g". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:28.480385 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.480347    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8tqd5\" (UniqueName: \"kubernetes.io/projected/bca2dabc-94e2-4875-ba3f-fa9332afe8dc-kube-api-access-8tqd5\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:28.480385 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.480378    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/e5ae7b61-cd5f-4a38-b038-28aaa6991eb7-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:28.480385 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.480387    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/bca2dabc-94e2-4875-ba3f-fa9332afe8dc-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:28.480631 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.480396    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wzv9g\" (UniqueName: \"kubernetes.io/projected/e5ae7b61-cd5f-4a38-b038-28aaa6991eb7-kube-api-access-wzv9g\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:28.552651 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.552573    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:28.999874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.999842    4456 generic.go:356] "Generic (PLEG): container finished" podID="e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" containerID="09f75adb956ba8d1b347af752e4d6d17af18d67cc64d9c62d56a97f64f5dde8e" exitCode=0
Nov 02 18:13:28.999874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.999873    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"
Nov 02 18:13:29.000287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.999917    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j" event={"ID":"e5ae7b61-cd5f-4a38-b038-28aaa6991eb7","Type":"ContainerDied","Data":"09f75adb956ba8d1b347af752e4d6d17af18d67cc64d9c62d56a97f64f5dde8e"}
Nov 02 18:13:29.000287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.999973    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j" event={"ID":"e5ae7b61-cd5f-4a38-b038-28aaa6991eb7","Type":"ContainerDied","Data":"5a4683ca1335bb45dee0f2972033c98eb26ce6dc1e172bf2a343ed5d8e4bc16c"}
Nov 02 18:13:29.000287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:28.999998    4456 scope.go:122] "RemoveContainer" containerID="09f75adb956ba8d1b347af752e4d6d17af18d67cc64d9c62d56a97f64f5dde8e"
Nov 02 18:13:29.001711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.001690    4456 generic.go:356] "Generic (PLEG): container finished" podID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" containerID="7c1de68e597eb58fefb3cd933a0ac134632987153b7d92161a06b331b295e194" exitCode=0
Nov 02 18:13:29.001808 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.001721    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" event={"ID":"bca2dabc-94e2-4875-ba3f-fa9332afe8dc","Type":"ContainerDied","Data":"7c1de68e597eb58fefb3cd933a0ac134632987153b7d92161a06b331b295e194"}
Nov 02 18:13:29.001808 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.001735    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"
Nov 02 18:13:29.001808 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.001753    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj" event={"ID":"bca2dabc-94e2-4875-ba3f-fa9332afe8dc","Type":"ContainerDied","Data":"8ef77d511cfd1d75eff7c29a150cd6acf17c03b9711c64acffc4f36a9e09d350"}
Nov 02 18:13:29.011494 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.011477    4456 scope.go:122] "RemoveContainer" containerID="09f75adb956ba8d1b347af752e4d6d17af18d67cc64d9c62d56a97f64f5dde8e"
Nov 02 18:13:29.011796 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:29.011773    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09f75adb956ba8d1b347af752e4d6d17af18d67cc64d9c62d56a97f64f5dde8e\": container with ID starting with 09f75adb956ba8d1b347af752e4d6d17af18d67cc64d9c62d56a97f64f5dde8e not found: ID does not exist" containerID="09f75adb956ba8d1b347af752e4d6d17af18d67cc64d9c62d56a97f64f5dde8e"
Nov 02 18:13:29.011898 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.011806    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09f75adb956ba8d1b347af752e4d6d17af18d67cc64d9c62d56a97f64f5dde8e"} err="failed to get container status \"09f75adb956ba8d1b347af752e4d6d17af18d67cc64d9c62d56a97f64f5dde8e\": rpc error: code = NotFound desc = could not find container \"09f75adb956ba8d1b347af752e4d6d17af18d67cc64d9c62d56a97f64f5dde8e\": container with ID starting with 09f75adb956ba8d1b347af752e4d6d17af18d67cc64d9c62d56a97f64f5dde8e not found: ID does not exist"
Nov 02 18:13:29.011898 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.011849    4456 scope.go:122] "RemoveContainer" containerID="7c1de68e597eb58fefb3cd933a0ac134632987153b7d92161a06b331b295e194"
Nov 02 18:13:29.025480 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.025454    4456 scope.go:122] "RemoveContainer" containerID="7c1de68e597eb58fefb3cd933a0ac134632987153b7d92161a06b331b295e194"
Nov 02 18:13:29.025777 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:29.025759    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"7c1de68e597eb58fefb3cd933a0ac134632987153b7d92161a06b331b295e194\": container with ID starting with 7c1de68e597eb58fefb3cd933a0ac134632987153b7d92161a06b331b295e194 not found: ID does not exist" containerID="7c1de68e597eb58fefb3cd933a0ac134632987153b7d92161a06b331b295e194"
Nov 02 18:13:29.025891 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.025785    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"7c1de68e597eb58fefb3cd933a0ac134632987153b7d92161a06b331b295e194"} err="failed to get container status \"7c1de68e597eb58fefb3cd933a0ac134632987153b7d92161a06b331b295e194\": rpc error: code = NotFound desc = could not find container \"7c1de68e597eb58fefb3cd933a0ac134632987153b7d92161a06b331b295e194\": container with ID starting with 7c1de68e597eb58fefb3cd933a0ac134632987153b7d92161a06b331b295e194 not found: ID does not exist"
Nov 02 18:13:29.057651 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.057625    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"]
Nov 02 18:13:29.068537 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.068514    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-w4-2899ab-64648c4db5-d7pnj"]
Nov 02 18:13:29.096493 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.096446    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"]
Nov 02 18:13:29.099913 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.099888    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-worker-w4-2899ab-75d49dc7d7-mjh5j"]
Nov 02 18:13:29.535833 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.535777    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main" containerID="cri-o://d2c0ed473e942d6f113bde4b458bb80e06031b1d79f122c6822f5c1c0be919b4" gracePeriod=45
Nov 02 18:13:29.541081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.541007    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main" containerID="cri-o://a3d1aa796ad6acbcdaf08b6699b2eaf2cfb7eaf3f65b1073198205e1ca33cd12" gracePeriod=45
Nov 02 18:13:29.554924 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.554887    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main" containerID="cri-o://dee185d86f1b035d22e9bbaa40ba82863efde6c7b75aef6a674103522b0c63a9" gracePeriod=45
Nov 02 18:13:29.613463 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.613423    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" path="/var/lib/kubelet/pods/bca2dabc-94e2-4875-ba3f-fa9332afe8dc/volumes"
Nov 02 18:13:29.613934 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.613914    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" path="/var/lib/kubelet/pods/e5ae7b61-cd5f-4a38-b038-28aaa6991eb7/volumes"
Nov 02 18:13:29.823579 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.823534    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 18:13:29.838270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.838244    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 18:13:29.864390 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.864369    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 18:13:29.896057 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896031    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-model-cache\") pod \"0c6cec58-b8d9-4856-9446-811407a747ab\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") "
Nov 02 18:13:29.896225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896063    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-home\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-home\") pod \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") "
Nov 02 18:13:29.896225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896080    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-dshm\") pod \"0874d0e8-5883-477d-b64a-d455934cfc90\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") "
Nov 02 18:13:29.896225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896156    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0874d0e8-5883-477d-b64a-d455934cfc90-kube-api-access-vfwdc\" (UniqueName: \"kubernetes.io/projected/0874d0e8-5883-477d-b64a-d455934cfc90-kube-api-access-vfwdc\") pod \"0874d0e8-5883-477d-b64a-d455934cfc90\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") "
Nov 02 18:13:29.896225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896191    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f563607b-3be4-4194-aa10-81e8bf2cbf6e-kube-api-access-69cwr\" (UniqueName: \"kubernetes.io/projected/f563607b-3be4-4194-aa10-81e8bf2cbf6e-kube-api-access-69cwr\") pod \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") "
Nov 02 18:13:29.896443 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896367    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/0c6cec58-b8d9-4856-9446-811407a747ab-kube-api-access-sns45\" (UniqueName: \"kubernetes.io/projected/0c6cec58-b8d9-4856-9446-811407a747ab-kube-api-access-sns45\") pod \"0c6cec58-b8d9-4856-9446-811407a747ab\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") "
Nov 02 18:13:29.896443 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896421    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-tmp-dir\") pod \"0874d0e8-5883-477d-b64a-d455934cfc90\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") "
Nov 02 18:13:29.896443 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896366    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-model-cache" pod "0c6cec58-b8d9-4856-9446-811407a747ab" (UID: "0c6cec58-b8d9-4856-9446-811407a747ab"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:29.896602 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896423    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-home" pod "f563607b-3be4-4194-aa10-81e8bf2cbf6e" (UID: "f563607b-3be4-4194-aa10-81e8bf2cbf6e"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:29.896602 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896492    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-model-cache\") pod \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") "
Nov 02 18:13:29.896602 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896541    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0874d0e8-5883-477d-b64a-d455934cfc90-tls-certs\" (UniqueName: \"kubernetes.io/secret/0874d0e8-5883-477d-b64a-d455934cfc90-tls-certs\") pod \"0874d0e8-5883-477d-b64a-d455934cfc90\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") "
Nov 02 18:13:29.896602 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896596    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/0c6cec58-b8d9-4856-9446-811407a747ab-tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6cec58-b8d9-4856-9446-811407a747ab-tls-certs\") pod \"0c6cec58-b8d9-4856-9446-811407a747ab\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") "
Nov 02 18:13:29.896793 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896651    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-model-cache\") pod \"0874d0e8-5883-477d-b64a-d455934cfc90\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") "
Nov 02 18:13:29.896793 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896683    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/f563607b-3be4-4194-aa10-81e8bf2cbf6e-tls-certs\" (UniqueName: \"kubernetes.io/secret/f563607b-3be4-4194-aa10-81e8bf2cbf6e-tls-certs\") pod \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") "
Nov 02 18:13:29.896793 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896711    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-tmp-dir\") pod \"0c6cec58-b8d9-4856-9446-811407a747ab\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") "
Nov 02 18:13:29.896793 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896738    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-home\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-home\") pod \"0c6cec58-b8d9-4856-9446-811407a747ab\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") "
Nov 02 18:13:29.896793 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896768    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-tmp-dir\") pod \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") "
Nov 02 18:13:29.897077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896927    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-dshm\") pod \"0c6cec58-b8d9-4856-9446-811407a747ab\" (UID: \"0c6cec58-b8d9-4856-9446-811407a747ab\") "
Nov 02 18:13:29.897077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896961    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-home\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-home\") pod \"0874d0e8-5883-477d-b64a-d455934cfc90\" (UID: \"0874d0e8-5883-477d-b64a-d455934cfc90\") "
Nov 02 18:13:29.897177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.897051    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-dshm\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-dshm\") pod \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\" (UID: \"f563607b-3be4-4194-aa10-81e8bf2cbf6e\") "
Nov 02 18:13:29.897432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896737    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-tmp-dir" pod "0874d0e8-5883-477d-b64a-d455934cfc90" (UID: "0874d0e8-5883-477d-b64a-d455934cfc90"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:29.897506 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.896892    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-model-cache" pod "f563607b-3be4-4194-aa10-81e8bf2cbf6e" (UID: "f563607b-3be4-4194-aa10-81e8bf2cbf6e"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:29.898831 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.897886    4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-model-cache\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.898831 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.897914    4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-home\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.898831 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.897929    4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-tmp-dir\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.898831 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.897943    4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-model-cache\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.898831 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.897960    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-model-cache" pod "0874d0e8-5883-477d-b64a-d455934cfc90" (UID: "0874d0e8-5883-477d-b64a-d455934cfc90"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:29.898831 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.898708    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-tmp-dir" pod "f563607b-3be4-4194-aa10-81e8bf2cbf6e" (UID: "f563607b-3be4-4194-aa10-81e8bf2cbf6e"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:29.898831 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.898791    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-tmp-dir" pod "0c6cec58-b8d9-4856-9446-811407a747ab" (UID: "0c6cec58-b8d9-4856-9446-811407a747ab"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:29.899204 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.898834    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-home" pod "0874d0e8-5883-477d-b64a-d455934cfc90" (UID: "0874d0e8-5883-477d-b64a-d455934cfc90"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:29.899204 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.898892    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-home" pod "0c6cec58-b8d9-4856-9446-811407a747ab" (UID: "0c6cec58-b8d9-4856-9446-811407a747ab"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:29.901661 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.901635    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0874d0e8-5883-477d-b64a-d455934cfc90-kube-api-access-vfwdc" pod "0874d0e8-5883-477d-b64a-d455934cfc90" (UID: "0874d0e8-5883-477d-b64a-d455934cfc90"). InnerVolumeSpecName "kube-api-access-vfwdc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:29.901762 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.901668    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0c6cec58-b8d9-4856-9446-811407a747ab-tls-certs" pod "0c6cec58-b8d9-4856-9446-811407a747ab" (UID: "0c6cec58-b8d9-4856-9446-811407a747ab"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:29.901762 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.901672    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/0c6cec58-b8d9-4856-9446-811407a747ab-kube-api-access-sns45" pod "0c6cec58-b8d9-4856-9446-811407a747ab" (UID: "0c6cec58-b8d9-4856-9446-811407a747ab"). InnerVolumeSpecName "kube-api-access-sns45". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:29.901951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.901876    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-dshm" pod "f563607b-3be4-4194-aa10-81e8bf2cbf6e" (UID: "f563607b-3be4-4194-aa10-81e8bf2cbf6e"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:29.902041 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.902019    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/0874d0e8-5883-477d-b64a-d455934cfc90-tls-certs" pod "0874d0e8-5883-477d-b64a-d455934cfc90" (UID: "0874d0e8-5883-477d-b64a-d455934cfc90"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:29.902122 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.902021    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-dshm" pod "0874d0e8-5883-477d-b64a-d455934cfc90" (UID: "0874d0e8-5883-477d-b64a-d455934cfc90"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:29.902122 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.902097    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f563607b-3be4-4194-aa10-81e8bf2cbf6e-kube-api-access-69cwr" pod "f563607b-3be4-4194-aa10-81e8bf2cbf6e" (UID: "f563607b-3be4-4194-aa10-81e8bf2cbf6e"). InnerVolumeSpecName "kube-api-access-69cwr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:29.902254 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.902238    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/f563607b-3be4-4194-aa10-81e8bf2cbf6e-tls-certs" pod "f563607b-3be4-4194-aa10-81e8bf2cbf6e" (UID: "f563607b-3be4-4194-aa10-81e8bf2cbf6e"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:29.902306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.902282    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-dshm" pod "0c6cec58-b8d9-4856-9446-811407a747ab" (UID: "0c6cec58-b8d9-4856-9446-811407a747ab"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:29.994481 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.994436    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"]
Nov 02 18:13:29.994998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.994974    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" containerName="payload-pre-processing"
Nov 02 18:13:29.994998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.994998    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" containerName="payload-pre-processing"
Nov 02 18:13:29.995183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995017    4456 cpu_manager.go:414] "containerMap: removing container" podUID="887b756a-8d5e-4510-9f48-0223365a0543" containerName="revoke-keys"
Nov 02 18:13:29.995183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995024    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="887b756a-8d5e-4510-9f48-0223365a0543" containerName="revoke-keys"
Nov 02 18:13:29.995183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995054    4456 cpu_manager.go:414] "containerMap: removing container" podUID="cc0bc138-76e3-41ca-83ac-c32c5a6770ee" containerName="revoke-keys"
Nov 02 18:13:29.995183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995063    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="cc0bc138-76e3-41ca-83ac-c32c5a6770ee" containerName="revoke-keys"
Nov 02 18:13:29.995183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995080    4456 cpu_manager.go:414] "containerMap: removing container" podUID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" containerName="payload-processing"
Nov 02 18:13:29.995183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995088    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" containerName="payload-processing"
Nov 02 18:13:29.995183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995099    4456 cpu_manager.go:414] "containerMap: removing container" podUID="ac13cb59-6208-4268-818f-205e6105789b" containerName="revoke-keys"
Nov 02 18:13:29.995183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995104    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ac13cb59-6208-4268-818f-205e6105789b" containerName="revoke-keys"
Nov 02 18:13:29.995183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995116    4456 cpu_manager.go:414] "containerMap: removing container" podUID="18e153e5-bd9d-4a00-8c94-75f6a53af87c" containerName="revoke-keys"
Nov 02 18:13:29.995183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995121    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="18e153e5-bd9d-4a00-8c94-75f6a53af87c" containerName="revoke-keys"
Nov 02 18:13:29.995183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995132    4456 cpu_manager.go:414] "containerMap: removing container" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main"
Nov 02 18:13:29.995183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995137    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main"
Nov 02 18:13:29.995800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995152    4456 cpu_manager.go:414] "containerMap: removing container" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main"
Nov 02 18:13:29.995800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995160    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main"
Nov 02 18:13:29.995800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995175    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main"
Nov 02 18:13:29.995800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995181    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main"
Nov 02 18:13:29.995800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995262    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="cc0bc138-76e3-41ca-83ac-c32c5a6770ee" containerName="revoke-keys"
Nov 02 18:13:29.995800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995273    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="bca2dabc-94e2-4875-ba3f-fa9332afe8dc" containerName="payload-processing"
Nov 02 18:13:29.995800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995281    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ac13cb59-6208-4268-818f-205e6105789b" containerName="revoke-keys"
Nov 02 18:13:29.995800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995289    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="18e153e5-bd9d-4a00-8c94-75f6a53af87c" containerName="revoke-keys"
Nov 02 18:13:29.995800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995296    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" containerName="main"
Nov 02 18:13:29.995800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995302    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" containerName="main"
Nov 02 18:13:29.995800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995308    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerName="main"
Nov 02 18:13:29.995800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995317    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e5ae7b61-cd5f-4a38-b038-28aaa6991eb7" containerName="payload-pre-processing"
Nov 02 18:13:29.996448 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.995334    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="887b756a-8d5e-4510-9f48-0223365a0543" containerName="revoke-keys"
Nov 02 18:13:29.998956 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.998926    4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0874d0e8-5883-477d-b64a-d455934cfc90-tls-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.999050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.998962    4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/0c6cec58-b8d9-4856-9446-811407a747ab-tls-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.999050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.998978    4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-model-cache\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.999050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.998992    4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/f563607b-3be4-4194-aa10-81e8bf2cbf6e-tls-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.999050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.999006    4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-tmp-dir\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.999050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.999019    4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-home\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.999050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.999033    4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-tmp-dir\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.999050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.999046    4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0c6cec58-b8d9-4856-9446-811407a747ab-dshm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.999307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.999060    4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-home\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.999307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.999074    4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/f563607b-3be4-4194-aa10-81e8bf2cbf6e-dshm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.999307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.999087    4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/0874d0e8-5883-477d-b64a-d455934cfc90-dshm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.999307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.999102    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-vfwdc\" (UniqueName: \"kubernetes.io/projected/0874d0e8-5883-477d-b64a-d455934cfc90-kube-api-access-vfwdc\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.999307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.999117    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-69cwr\" (UniqueName: \"kubernetes.io/projected/f563607b-3be4-4194-aa10-81e8bf2cbf6e-kube-api-access-69cwr\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:29.999307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:29.999133    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-sns45\" (UniqueName: \"kubernetes.io/projected/0c6cec58-b8d9-4856-9446-811407a747ab-kube-api-access-sns45\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:30.000120 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.000099    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"
Nov 02 18:13:30.014476 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.014446    4456 generic.go:356] "Generic (PLEG): container finished" podID="0874d0e8-5883-477d-b64a-d455934cfc90" containerID="d2c0ed473e942d6f113bde4b458bb80e06031b1d79f122c6822f5c1c0be919b4" exitCode=0
Nov 02 18:13:30.014645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.014625    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" event={"ID":"0874d0e8-5883-477d-b64a-d455934cfc90","Type":"ContainerDied","Data":"d2c0ed473e942d6f113bde4b458bb80e06031b1d79f122c6822f5c1c0be919b4"}
Nov 02 18:13:30.014716 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.014679    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz" event={"ID":"0874d0e8-5883-477d-b64a-d455934cfc90","Type":"ContainerDied","Data":"a3900d4e7b8bd64d510d5e5a0f6580e0160fb9f87f5221cc11fdbf0df3354938"}
Nov 02 18:13:30.014716 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.014703    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"]
Nov 02 18:13:30.016990 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.016854    4456 scope.go:122] "RemoveContainer" containerID="d2c0ed473e942d6f113bde4b458bb80e06031b1d79f122c6822f5c1c0be919b4"
Nov 02 18:13:30.017115 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.017093    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"
Nov 02 18:13:30.019650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.018684    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:13:30.019650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.018722    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:13:30.019650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.018763    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:13:30.019650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.018895    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:13:30.021798 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.021777    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"]
Nov 02 18:13:30.021957 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.021900    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"
Nov 02 18:13:30.024618 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.024589    4456 generic.go:356] "Generic (PLEG): container finished" podID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" containerID="a3d1aa796ad6acbcdaf08b6699b2eaf2cfb7eaf3f65b1073198205e1ca33cd12" exitCode=0
Nov 02 18:13:30.031832 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.024948    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"
Nov 02 18:13:30.031832 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.025080    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" event={"ID":"f563607b-3be4-4194-aa10-81e8bf2cbf6e","Type":"ContainerDied","Data":"a3d1aa796ad6acbcdaf08b6699b2eaf2cfb7eaf3f65b1073198205e1ca33cd12"}
Nov 02 18:13:30.031832 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.025146    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq" event={"ID":"f563607b-3be4-4194-aa10-81e8bf2cbf6e","Type":"ContainerDied","Data":"abb99a7cd1f8a60cc21779e528634a40031055d969f794a8c36d0090bcb1ff3e"}
Nov 02 18:13:30.031832 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.027149    4456 generic.go:356] "Generic (PLEG): container finished" podID="0c6cec58-b8d9-4856-9446-811407a747ab" containerID="dee185d86f1b035d22e9bbaa40ba82863efde6c7b75aef6a674103522b0c63a9" exitCode=0
Nov 02 18:13:30.031832 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.027219    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" event={"ID":"0c6cec58-b8d9-4856-9446-811407a747ab","Type":"ContainerDied","Data":"dee185d86f1b035d22e9bbaa40ba82863efde6c7b75aef6a674103522b0c63a9"}
Nov 02 18:13:30.031832 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.027255    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5" event={"ID":"0c6cec58-b8d9-4856-9446-811407a747ab","Type":"ContainerDied","Data":"01a5b69e8c051a84622bfd5f598c65f6ed739617d900941ec81338e51762888e"}
Nov 02 18:13:30.031832 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.027336    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"
Nov 02 18:13:30.042433 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.042403    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"]
Nov 02 18:13:30.047923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.047895    4456 scope.go:122] "RemoveContainer" containerID="d2c0ed473e942d6f113bde4b458bb80e06031b1d79f122c6822f5c1c0be919b4"
Nov 02 18:13:30.048442 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:30.048343    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d2c0ed473e942d6f113bde4b458bb80e06031b1d79f122c6822f5c1c0be919b4\": container with ID starting with d2c0ed473e942d6f113bde4b458bb80e06031b1d79f122c6822f5c1c0be919b4 not found: ID does not exist" containerID="d2c0ed473e942d6f113bde4b458bb80e06031b1d79f122c6822f5c1c0be919b4"
Nov 02 18:13:30.048442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.048389    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d2c0ed473e942d6f113bde4b458bb80e06031b1d79f122c6822f5c1c0be919b4"} err="failed to get container status \"d2c0ed473e942d6f113bde4b458bb80e06031b1d79f122c6822f5c1c0be919b4\": rpc error: code = NotFound desc = could not find container \"d2c0ed473e942d6f113bde4b458bb80e06031b1d79f122c6822f5c1c0be919b4\": container with ID starting with d2c0ed473e942d6f113bde4b458bb80e06031b1d79f122c6822f5c1c0be919b4 not found: ID does not exist"
Nov 02 18:13:30.048442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.048415    4456 scope.go:122] "RemoveContainer" containerID="a3d1aa796ad6acbcdaf08b6699b2eaf2cfb7eaf3f65b1073198205e1ca33cd12"
Nov 02 18:13:30.066186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.066160    4456 scope.go:122] "RemoveContainer" containerID="a3d1aa796ad6acbcdaf08b6699b2eaf2cfb7eaf3f65b1073198205e1ca33cd12"
Nov 02 18:13:30.067300 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:30.067276    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a3d1aa796ad6acbcdaf08b6699b2eaf2cfb7eaf3f65b1073198205e1ca33cd12\": container with ID starting with a3d1aa796ad6acbcdaf08b6699b2eaf2cfb7eaf3f65b1073198205e1ca33cd12 not found: ID does not exist" containerID="a3d1aa796ad6acbcdaf08b6699b2eaf2cfb7eaf3f65b1073198205e1ca33cd12"
Nov 02 18:13:30.067394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.067309    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a3d1aa796ad6acbcdaf08b6699b2eaf2cfb7eaf3f65b1073198205e1ca33cd12"} err="failed to get container status \"a3d1aa796ad6acbcdaf08b6699b2eaf2cfb7eaf3f65b1073198205e1ca33cd12\": rpc error: code = NotFound desc = could not find container \"a3d1aa796ad6acbcdaf08b6699b2eaf2cfb7eaf3f65b1073198205e1ca33cd12\": container with ID starting with a3d1aa796ad6acbcdaf08b6699b2eaf2cfb7eaf3f65b1073198205e1ca33cd12 not found: ID does not exist"
Nov 02 18:13:30.067394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.067335    4456 scope.go:122] "RemoveContainer" containerID="dee185d86f1b035d22e9bbaa40ba82863efde6c7b75aef6a674103522b0c63a9"
Nov 02 18:13:30.080446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.080169    4456 scope.go:122] "RemoveContainer" containerID="dee185d86f1b035d22e9bbaa40ba82863efde6c7b75aef6a674103522b0c63a9"
Nov 02 18:13:30.080554 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:30.080535    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"dee185d86f1b035d22e9bbaa40ba82863efde6c7b75aef6a674103522b0c63a9\": container with ID starting with dee185d86f1b035d22e9bbaa40ba82863efde6c7b75aef6a674103522b0c63a9 not found: ID does not exist" containerID="dee185d86f1b035d22e9bbaa40ba82863efde6c7b75aef6a674103522b0c63a9"
Nov 02 18:13:30.080633 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.080585    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"dee185d86f1b035d22e9bbaa40ba82863efde6c7b75aef6a674103522b0c63a9"} err="failed to get container status \"dee185d86f1b035d22e9bbaa40ba82863efde6c7b75aef6a674103522b0c63a9\": rpc error: code = NotFound desc = could not find container \"dee185d86f1b035d22e9bbaa40ba82863efde6c7b75aef6a674103522b0c63a9\": container with ID starting with dee185d86f1b035d22e9bbaa40ba82863efde6c7b75aef6a674103522b0c63a9 not found: ID does not exist"
Nov 02 18:13:30.087080 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.087056    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"]
Nov 02 18:13:30.092378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.092347    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"
Nov 02 18:13:30.099809 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.099791    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-75fnb\" (UniqueName: \"kubernetes.io/projected/7e85b538-1702-463b-adc2-23276a23367e-kube-api-access-75fnb\") pod \"maas-api-revoke-sub-e426cc5f-pp2qt\" (UID: \"7e85b538-1702-463b-adc2-23276a23367e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"
Nov 02 18:13:30.100076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.099947    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e85b538-1702-463b-adc2-23276a23367e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e426cc5f-pp2qt\" (UID: \"7e85b538-1702-463b-adc2-23276a23367e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"
Nov 02 18:13:30.100076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.100011    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e85b538-1702-463b-adc2-23276a23367e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e426cc5f-pp2qt\" (UID: \"7e85b538-1702-463b-adc2-23276a23367e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"
Nov 02 18:13:30.100076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.100050    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc67f3e-e8e3-4d57-859c-abb8561601a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bf0d7a5-dk2dp\" (UID: \"6cc67f3e-e8e3-4d57-859c-abb8561601a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"
Nov 02 18:13:30.100244 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.100139    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc67f3e-e8e3-4d57-859c-abb8561601a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bf0d7a5-dk2dp\" (UID: \"6cc67f3e-e8e3-4d57-859c-abb8561601a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"
Nov 02 18:13:30.100244 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.100206    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-c4xnr\" (UniqueName: \"kubernetes.io/projected/6cc67f3e-e8e3-4d57-859c-abb8561601a0-kube-api-access-c4xnr\") pod \"maas-api-revoke-sub-6bf0d7a5-dk2dp\" (UID: \"6cc67f3e-e8e3-4d57-859c-abb8561601a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"
Nov 02 18:13:30.109733 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.109709    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"]
Nov 02 18:13:30.114859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.113443    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"]
Nov 02 18:13:30.117699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.117672    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"
Nov 02 18:13:30.142107 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.142076    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"]
Nov 02 18:13:30.201682 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.201642    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e85b538-1702-463b-adc2-23276a23367e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e426cc5f-pp2qt\" (UID: \"7e85b538-1702-463b-adc2-23276a23367e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"
Nov 02 18:13:30.201869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.201710    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e85b538-1702-463b-adc2-23276a23367e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e426cc5f-pp2qt\" (UID: \"7e85b538-1702-463b-adc2-23276a23367e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"
Nov 02 18:13:30.201869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.201747    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc67f3e-e8e3-4d57-859c-abb8561601a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bf0d7a5-dk2dp\" (UID: \"6cc67f3e-e8e3-4d57-859c-abb8561601a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"
Nov 02 18:13:30.201869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.201786    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3af94642-hjdmc\" (UID: \"1d81d958-89bb-4f35-8c6c-d7a09d2507ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"
Nov 02 18:13:30.202022 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.201890    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc67f3e-e8e3-4d57-859c-abb8561601a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bf0d7a5-dk2dp\" (UID: \"6cc67f3e-e8e3-4d57-859c-abb8561601a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"
Nov 02 18:13:30.202242 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.201941    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-c4xnr\" (UniqueName: \"kubernetes.io/projected/6cc67f3e-e8e3-4d57-859c-abb8561601a0-kube-api-access-c4xnr\") pod \"maas-api-revoke-sub-6bf0d7a5-dk2dp\" (UID: \"6cc67f3e-e8e3-4d57-859c-abb8561601a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"
Nov 02 18:13:30.202311 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.202288    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95809ee-3d8e-4197-9792-342b1c7a4820-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1871ff23-sthwn\" (UID: \"d95809ee-3d8e-4197-9792-342b1c7a4820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"
Nov 02 18:13:30.202374 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.202337    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3af94642-hjdmc\" (UID: \"1d81d958-89bb-4f35-8c6c-d7a09d2507ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"
Nov 02 18:13:30.202468 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.202451    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jgfw6\" (UniqueName: \"kubernetes.io/projected/d95809ee-3d8e-4197-9792-342b1c7a4820-kube-api-access-jgfw6\") pod \"maas-api-revoke-sub-1871ff23-sthwn\" (UID: \"d95809ee-3d8e-4197-9792-342b1c7a4820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"
Nov 02 18:13:30.202713 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.202478    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-ph9z8\" (UniqueName: \"kubernetes.io/projected/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-kube-api-access-ph9z8\") pod \"maas-api-revoke-sub-3af94642-hjdmc\" (UID: \"1d81d958-89bb-4f35-8c6c-d7a09d2507ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"
Nov 02 18:13:30.202713 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.202514    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-75fnb\" (UniqueName: \"kubernetes.io/projected/7e85b538-1702-463b-adc2-23276a23367e-kube-api-access-75fnb\") pod \"maas-api-revoke-sub-e426cc5f-pp2qt\" (UID: \"7e85b538-1702-463b-adc2-23276a23367e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"
Nov 02 18:13:30.202713 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.202556    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95809ee-3d8e-4197-9792-342b1c7a4820-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1871ff23-sthwn\" (UID: \"d95809ee-3d8e-4197-9792-342b1c7a4820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"
Nov 02 18:13:30.202713 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.202656    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc67f3e-e8e3-4d57-859c-abb8561601a0-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-6bf0d7a5-dk2dp\" (UID: \"6cc67f3e-e8e3-4d57-859c-abb8561601a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"
Nov 02 18:13:30.203077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.203052    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e85b538-1702-463b-adc2-23276a23367e-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e426cc5f-pp2qt\" (UID: \"7e85b538-1702-463b-adc2-23276a23367e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"
Nov 02 18:13:30.204645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.204620    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e85b538-1702-463b-adc2-23276a23367e-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e426cc5f-pp2qt\" (UID: \"7e85b538-1702-463b-adc2-23276a23367e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"
Nov 02 18:13:30.204755 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.204620    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc67f3e-e8e3-4d57-859c-abb8561601a0-maas-api-xks-ca\") pod \"maas-api-revoke-sub-6bf0d7a5-dk2dp\" (UID: \"6cc67f3e-e8e3-4d57-859c-abb8561601a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"
Nov 02 18:13:30.229165 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.229133    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-c4xnr\" (UniqueName: \"kubernetes.io/projected/6cc67f3e-e8e3-4d57-859c-abb8561601a0-kube-api-access-c4xnr\") pod \"maas-api-revoke-sub-6bf0d7a5-dk2dp\" (UID: \"6cc67f3e-e8e3-4d57-859c-abb8561601a0\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"
Nov 02 18:13:30.230504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.230478    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-75fnb\" (UniqueName: \"kubernetes.io/projected/7e85b538-1702-463b-adc2-23276a23367e-kube-api-access-75fnb\") pod \"maas-api-revoke-sub-e426cc5f-pp2qt\" (UID: \"7e85b538-1702-463b-adc2-23276a23367e\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"
Nov 02 18:13:30.269652 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.269625    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"]
Nov 02 18:13:30.273894 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.273870    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/simulator-w3-5eedab-kserve-795c4575bb-f25x5"]
Nov 02 18:13:30.303537 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.303512    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jgfw6\" (UniqueName: \"kubernetes.io/projected/d95809ee-3d8e-4197-9792-342b1c7a4820-kube-api-access-jgfw6\") pod \"maas-api-revoke-sub-1871ff23-sthwn\" (UID: \"d95809ee-3d8e-4197-9792-342b1c7a4820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"
Nov 02 18:13:30.303537 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.303542    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-ph9z8\" (UniqueName: \"kubernetes.io/projected/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-kube-api-access-ph9z8\") pod \"maas-api-revoke-sub-3af94642-hjdmc\" (UID: \"1d81d958-89bb-4f35-8c6c-d7a09d2507ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"
Nov 02 18:13:30.303742 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.303581    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95809ee-3d8e-4197-9792-342b1c7a4820-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1871ff23-sthwn\" (UID: \"d95809ee-3d8e-4197-9792-342b1c7a4820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"
Nov 02 18:13:30.303742 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.303637    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3af94642-hjdmc\" (UID: \"1d81d958-89bb-4f35-8c6c-d7a09d2507ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"
Nov 02 18:13:30.303742 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.303715    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95809ee-3d8e-4197-9792-342b1c7a4820-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1871ff23-sthwn\" (UID: \"d95809ee-3d8e-4197-9792-342b1c7a4820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"
Nov 02 18:13:30.303918 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.303781    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3af94642-hjdmc\" (UID: \"1d81d958-89bb-4f35-8c6c-d7a09d2507ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"
Nov 02 18:13:30.304444 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.304407    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95809ee-3d8e-4197-9792-342b1c7a4820-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-1871ff23-sthwn\" (UID: \"d95809ee-3d8e-4197-9792-342b1c7a4820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"
Nov 02 18:13:30.304723 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.304539    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3af94642-hjdmc\" (UID: \"1d81d958-89bb-4f35-8c6c-d7a09d2507ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"
Nov 02 18:13:30.307101 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.307078    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95809ee-3d8e-4197-9792-342b1c7a4820-maas-api-xks-ca\") pod \"maas-api-revoke-sub-1871ff23-sthwn\" (UID: \"d95809ee-3d8e-4197-9792-342b1c7a4820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"
Nov 02 18:13:30.307396 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.307379    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3af94642-hjdmc\" (UID: \"1d81d958-89bb-4f35-8c6c-d7a09d2507ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"
Nov 02 18:13:30.310591 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.310522    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"]
Nov 02 18:13:30.315367 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.315343    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/distinct-w3-5eedab-kserve-856b8c7458-k2lgz"]
Nov 02 18:13:30.321009 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.320992    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"
Nov 02 18:13:30.330681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.330657    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jgfw6\" (UniqueName: \"kubernetes.io/projected/d95809ee-3d8e-4197-9792-342b1c7a4820-kube-api-access-jgfw6\") pod \"maas-api-revoke-sub-1871ff23-sthwn\" (UID: \"d95809ee-3d8e-4197-9792-342b1c7a4820\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"
Nov 02 18:13:30.330807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.330790    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-ph9z8\" (UniqueName: \"kubernetes.io/projected/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-kube-api-access-ph9z8\") pod \"maas-api-revoke-sub-3af94642-hjdmc\" (UID: \"1d81d958-89bb-4f35-8c6c-d7a09d2507ae\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"
Nov 02 18:13:30.351133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.351092    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"]
Nov 02 18:13:30.354651 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.354627    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w3-5eedab/premium-w3-5eedab-kserve-786db78999-f8lbq"]
Nov 02 18:13:30.369294 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.369269    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"
Nov 02 18:13:30.408247 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.408027    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"
Nov 02 18:13:30.432325 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.432287    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"
Nov 02 18:13:30.532393 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.532360    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:30.626282 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:13:30.626253    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6cc67f3e_e8e3_4d57_859c_abb8561601a0.slice/crio-bbf762b38cbe1d918c8eee9af0cc71c3c194c37bb4b7a7ed2e5e46bd6adf9f0a WatchSource:0}: Error finding container bbf762b38cbe1d918c8eee9af0cc71c3c194c37bb4b7a7ed2e5e46bd6adf9f0a: Status 404 returned error can't find the container with id bbf762b38cbe1d918c8eee9af0cc71c3c194c37bb4b7a7ed2e5e46bd6adf9f0a
Nov 02 18:13:30.627272 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.627239    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"]
Nov 02 18:13:30.685279 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:13:30.685244    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7e85b538_1702_463b_adc2_23276a23367e.slice/crio-4a19ff00dd1f78f27d3d9c8b5bcae664021510ee5806de74371bf3dd649d8e48 WatchSource:0}: Error finding container 4a19ff00dd1f78f27d3d9c8b5bcae664021510ee5806de74371bf3dd649d8e48: Status 404 returned error can't find the container with id 4a19ff00dd1f78f27d3d9c8b5bcae664021510ee5806de74371bf3dd649d8e48
Nov 02 18:13:30.685480 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.685424    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"]
Nov 02 18:13:30.728129 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.728106    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"]
Nov 02 18:13:30.729700 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:13:30.729680    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod1d81d958_89bb_4f35_8c6c_d7a09d2507ae.slice/crio-3ba6f59f2fd9e1d1e75d84ce16b58159389fa1f9872b11e0e91a09ed3a39497d WatchSource:0}: Error finding container 3ba6f59f2fd9e1d1e75d84ce16b58159389fa1f9872b11e0e91a09ed3a39497d: Status 404 returned error can't find the container with id 3ba6f59f2fd9e1d1e75d84ce16b58159389fa1f9872b11e0e91a09ed3a39497d
Nov 02 18:13:30.765734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:30.765705    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"]
Nov 02 18:13:30.766010 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:13:30.765989    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd95809ee_3d8e_4197_9792_342b1c7a4820.slice/crio-244bd0074db8ff678b031076d0db8e99a228c95dad6f1a6c9115504e11677d39 WatchSource:0}: Error finding container 244bd0074db8ff678b031076d0db8e99a228c95dad6f1a6c9115504e11677d39: Status 404 returned error can't find the container with id 244bd0074db8ff678b031076d0db8e99a228c95dad6f1a6c9115504e11677d39
Nov 02 18:13:31.036514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:31.036483    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp" event={"ID":"6cc67f3e-e8e3-4d57-859c-abb8561601a0","Type":"ContainerStarted","Data":"bbf762b38cbe1d918c8eee9af0cc71c3c194c37bb4b7a7ed2e5e46bd6adf9f0a"}
Nov 02 18:13:31.037806 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:31.037774    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc" event={"ID":"1d81d958-89bb-4f35-8c6c-d7a09d2507ae","Type":"ContainerStarted","Data":"3ba6f59f2fd9e1d1e75d84ce16b58159389fa1f9872b11e0e91a09ed3a39497d"}
Nov 02 18:13:31.040211 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:31.040182    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt" event={"ID":"7e85b538-1702-463b-adc2-23276a23367e","Type":"ContainerStarted","Data":"4a19ff00dd1f78f27d3d9c8b5bcae664021510ee5806de74371bf3dd649d8e48"}
Nov 02 18:13:31.042589 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:31.042561    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn" event={"ID":"d95809ee-3d8e-4197-9792-342b1c7a4820","Type":"ContainerStarted","Data":"244bd0074db8ff678b031076d0db8e99a228c95dad6f1a6c9115504e11677d39"}
Nov 02 18:13:31.614751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:31.614662    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0874d0e8-5883-477d-b64a-d455934cfc90" path="/var/lib/kubelet/pods/0874d0e8-5883-477d-b64a-d455934cfc90/volumes"
Nov 02 18:13:31.615218 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:31.615194    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="0c6cec58-b8d9-4856-9446-811407a747ab" path="/var/lib/kubelet/pods/0c6cec58-b8d9-4856-9446-811407a747ab/volumes"
Nov 02 18:13:31.615656 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:31.615637    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f563607b-3be4-4194-aa10-81e8bf2cbf6e" path="/var/lib/kubelet/pods/f563607b-3be4-4194-aa10-81e8bf2cbf6e/volumes"
Nov 02 18:13:31.807721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:31.807694    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:31.948506 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:31.948464    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:32.052972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:32.052933    4456 generic.go:356] "Generic (PLEG): container finished" podID="6cc67f3e-e8e3-4d57-859c-abb8561601a0" containerID="52616e876db9daa613fa24f0dd3b85b8ef0d8b80816a4fb006e00af3d5f01eb2" exitCode=0
Nov 02 18:13:32.053432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:32.052997    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp" event={"ID":"6cc67f3e-e8e3-4d57-859c-abb8561601a0","Type":"ContainerDied","Data":"52616e876db9daa613fa24f0dd3b85b8ef0d8b80816a4fb006e00af3d5f01eb2"}
Nov 02 18:13:32.054965 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:32.054929    4456 generic.go:356] "Generic (PLEG): container finished" podID="1d81d958-89bb-4f35-8c6c-d7a09d2507ae" containerID="aabfa3c98cb11dbc7499d020d1614e4c598229153040f81cd5c7880778664361" exitCode=0
Nov 02 18:13:32.055078 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:32.055008    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc" event={"ID":"1d81d958-89bb-4f35-8c6c-d7a09d2507ae","Type":"ContainerDied","Data":"aabfa3c98cb11dbc7499d020d1614e4c598229153040f81cd5c7880778664361"}
Nov 02 18:13:32.057057 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:32.057034    4456 generic.go:356] "Generic (PLEG): container finished" podID="7e85b538-1702-463b-adc2-23276a23367e" containerID="590887138fef93f2da3a68becf81f54c2a49fd6f83725ff4291142fd190e6540" exitCode=0
Nov 02 18:13:32.057183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:32.057109    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt" event={"ID":"7e85b538-1702-463b-adc2-23276a23367e","Type":"ContainerDied","Data":"590887138fef93f2da3a68becf81f54c2a49fd6f83725ff4291142fd190e6540"}
Nov 02 18:13:32.058997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:32.058972    4456 generic.go:356] "Generic (PLEG): container finished" podID="d95809ee-3d8e-4197-9792-342b1c7a4820" containerID="b10406aea9ff1dbe6171294ed32b2f6cc04d1bd3e9c15d758a72762280786288" exitCode=0
Nov 02 18:13:32.059103 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:32.059059    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn" event={"ID":"d95809ee-3d8e-4197-9792-342b1c7a4820","Type":"ContainerDied","Data":"b10406aea9ff1dbe6171294ed32b2f6cc04d1bd3e9c15d758a72762280786288"}
Nov 02 18:13:32.090016 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:32.089988    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:32.512109 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:32.512071    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:32.645217 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:32.645178    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"]
Nov 02 18:13:32.649719 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:32.649681    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w6-7df526-ztzs9"]
Nov 02 18:13:32.672991 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:32.672939    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:33.134451 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.134310    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:33.221654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.221626    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"
Nov 02 18:13:33.282795 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.282772    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"
Nov 02 18:13:33.319128 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.319072    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"
Nov 02 18:13:33.323773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.323748    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"
Nov 02 18:13:33.347214 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.347183    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-maas-api-openshift-ca\") pod \"1d81d958-89bb-4f35-8c6c-d7a09d2507ae\" (UID: \"1d81d958-89bb-4f35-8c6c-d7a09d2507ae\") "
Nov 02 18:13:33.347370 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.347261    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6cc67f3e-e8e3-4d57-859c-abb8561601a0-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc67f3e-e8e3-4d57-859c-abb8561601a0-maas-api-openshift-ca\") pod \"6cc67f3e-e8e3-4d57-859c-abb8561601a0\" (UID: \"6cc67f3e-e8e3-4d57-859c-abb8561601a0\") "
Nov 02 18:13:33.347442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.347390    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/6cc67f3e-e8e3-4d57-859c-abb8561601a0-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc67f3e-e8e3-4d57-859c-abb8561601a0-maas-api-xks-ca\") pod \"6cc67f3e-e8e3-4d57-859c-abb8561601a0\" (UID: \"6cc67f3e-e8e3-4d57-859c-abb8561601a0\") "
Nov 02 18:13:33.347488 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.347451    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6cc67f3e-e8e3-4d57-859c-abb8561601a0-kube-api-access-c4xnr\" (UniqueName: \"kubernetes.io/projected/6cc67f3e-e8e3-4d57-859c-abb8561601a0-kube-api-access-c4xnr\") pod \"6cc67f3e-e8e3-4d57-859c-abb8561601a0\" (UID: \"6cc67f3e-e8e3-4d57-859c-abb8561601a0\") "
Nov 02 18:13:33.347542 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.347524    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-maas-api-xks-ca\") pod \"1d81d958-89bb-4f35-8c6c-d7a09d2507ae\" (UID: \"1d81d958-89bb-4f35-8c6c-d7a09d2507ae\") "
Nov 02 18:13:33.347983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.347600    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-kube-api-access-ph9z8\" (UniqueName: \"kubernetes.io/projected/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-kube-api-access-ph9z8\") pod \"1d81d958-89bb-4f35-8c6c-d7a09d2507ae\" (UID: \"1d81d958-89bb-4f35-8c6c-d7a09d2507ae\") "
Nov 02 18:13:33.347983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.347599    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6cc67f3e-e8e3-4d57-859c-abb8561601a0-maas-api-openshift-ca" pod "6cc67f3e-e8e3-4d57-859c-abb8561601a0" (UID: "6cc67f3e-e8e3-4d57-859c-abb8561601a0"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:33.347983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.347603    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-maas-api-openshift-ca" pod "1d81d958-89bb-4f35-8c6c-d7a09d2507ae" (UID: "1d81d958-89bb-4f35-8c6c-d7a09d2507ae"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:33.348186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.348057    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:33.348186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.348081    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/6cc67f3e-e8e3-4d57-859c-abb8561601a0-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:33.351793 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.350700    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-maas-api-xks-ca" pod "1d81d958-89bb-4f35-8c6c-d7a09d2507ae" (UID: "1d81d958-89bb-4f35-8c6c-d7a09d2507ae"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:33.351793 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.350932    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-kube-api-access-ph9z8" pod "1d81d958-89bb-4f35-8c6c-d7a09d2507ae" (UID: "1d81d958-89bb-4f35-8c6c-d7a09d2507ae"). InnerVolumeSpecName "kube-api-access-ph9z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:33.352848 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.352782    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/6cc67f3e-e8e3-4d57-859c-abb8561601a0-maas-api-xks-ca" pod "6cc67f3e-e8e3-4d57-859c-abb8561601a0" (UID: "6cc67f3e-e8e3-4d57-859c-abb8561601a0"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:33.353109 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.353087    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6cc67f3e-e8e3-4d57-859c-abb8561601a0-kube-api-access-c4xnr" pod "6cc67f3e-e8e3-4d57-859c-abb8561601a0" (UID: "6cc67f3e-e8e3-4d57-859c-abb8561601a0"). InnerVolumeSpecName "kube-api-access-c4xnr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:33.448714 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.448682    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7e85b538-1702-463b-adc2-23276a23367e-kube-api-access-75fnb\" (UniqueName: \"kubernetes.io/projected/7e85b538-1702-463b-adc2-23276a23367e-kube-api-access-75fnb\") pod \"7e85b538-1702-463b-adc2-23276a23367e\" (UID: \"7e85b538-1702-463b-adc2-23276a23367e\") "
Nov 02 18:13:33.448889 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.448727    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7e85b538-1702-463b-adc2-23276a23367e-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e85b538-1702-463b-adc2-23276a23367e-maas-api-xks-ca\") pod \"7e85b538-1702-463b-adc2-23276a23367e\" (UID: \"7e85b538-1702-463b-adc2-23276a23367e\") "
Nov 02 18:13:33.448889 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.448772    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d95809ee-3d8e-4197-9792-342b1c7a4820-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95809ee-3d8e-4197-9792-342b1c7a4820-maas-api-openshift-ca\") pod \"d95809ee-3d8e-4197-9792-342b1c7a4820\" (UID: \"d95809ee-3d8e-4197-9792-342b1c7a4820\") "
Nov 02 18:13:33.448889 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.448797    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d95809ee-3d8e-4197-9792-342b1c7a4820-kube-api-access-jgfw6\" (UniqueName: \"kubernetes.io/projected/d95809ee-3d8e-4197-9792-342b1c7a4820-kube-api-access-jgfw6\") pod \"d95809ee-3d8e-4197-9792-342b1c7a4820\" (UID: \"d95809ee-3d8e-4197-9792-342b1c7a4820\") "
Nov 02 18:13:33.449050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.448908    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d95809ee-3d8e-4197-9792-342b1c7a4820-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95809ee-3d8e-4197-9792-342b1c7a4820-maas-api-xks-ca\") pod \"d95809ee-3d8e-4197-9792-342b1c7a4820\" (UID: \"d95809ee-3d8e-4197-9792-342b1c7a4820\") "
Nov 02 18:13:33.449050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.448944    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7e85b538-1702-463b-adc2-23276a23367e-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e85b538-1702-463b-adc2-23276a23367e-maas-api-openshift-ca\") pod \"7e85b538-1702-463b-adc2-23276a23367e\" (UID: \"7e85b538-1702-463b-adc2-23276a23367e\") "
Nov 02 18:13:33.449439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.449196    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d95809ee-3d8e-4197-9792-342b1c7a4820-maas-api-openshift-ca" pod "d95809ee-3d8e-4197-9792-342b1c7a4820" (UID: "d95809ee-3d8e-4197-9792-342b1c7a4820"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:33.449439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.449404    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/6cc67f3e-e8e3-4d57-859c-abb8561601a0-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:33.449439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.449427    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-c4xnr\" (UniqueName: \"kubernetes.io/projected/6cc67f3e-e8e3-4d57-859c-abb8561601a0-kube-api-access-c4xnr\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:33.449439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.449441    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d95809ee-3d8e-4197-9792-342b1c7a4820-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:33.449703 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.449458    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:33.449703 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.449471    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ph9z8\" (UniqueName: \"kubernetes.io/projected/1d81d958-89bb-4f35-8c6c-d7a09d2507ae-kube-api-access-ph9z8\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:33.449703 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.449534    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7e85b538-1702-463b-adc2-23276a23367e-maas-api-openshift-ca" pod "7e85b538-1702-463b-adc2-23276a23367e" (UID: "7e85b538-1702-463b-adc2-23276a23367e"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:33.451360 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.451334    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d95809ee-3d8e-4197-9792-342b1c7a4820-maas-api-xks-ca" pod "d95809ee-3d8e-4197-9792-342b1c7a4820" (UID: "d95809ee-3d8e-4197-9792-342b1c7a4820"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:33.451518 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.451499    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d95809ee-3d8e-4197-9792-342b1c7a4820-kube-api-access-jgfw6" pod "d95809ee-3d8e-4197-9792-342b1c7a4820" (UID: "d95809ee-3d8e-4197-9792-342b1c7a4820"). InnerVolumeSpecName "kube-api-access-jgfw6". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:33.451597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.451579    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7e85b538-1702-463b-adc2-23276a23367e-maas-api-xks-ca" pod "7e85b538-1702-463b-adc2-23276a23367e" (UID: "7e85b538-1702-463b-adc2-23276a23367e"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:33.451714 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.451699    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7e85b538-1702-463b-adc2-23276a23367e-kube-api-access-75fnb" pod "7e85b538-1702-463b-adc2-23276a23367e" (UID: "7e85b538-1702-463b-adc2-23276a23367e"). InnerVolumeSpecName "kube-api-access-75fnb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:33.550491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.550460    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jgfw6\" (UniqueName: \"kubernetes.io/projected/d95809ee-3d8e-4197-9792-342b1c7a4820-kube-api-access-jgfw6\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:33.550491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.550486    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d95809ee-3d8e-4197-9792-342b1c7a4820-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:33.550491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.550497    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7e85b538-1702-463b-adc2-23276a23367e-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:33.550703 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.550506    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-75fnb\" (UniqueName: \"kubernetes.io/projected/7e85b538-1702-463b-adc2-23276a23367e-kube-api-access-75fnb\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:33.550703 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.550515    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7e85b538-1702-463b-adc2-23276a23367e-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:33.615331 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:33.615258    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ea498d1b-474a-42c1-a576-2e0005a2a8c6" path="/var/lib/kubelet/pods/ea498d1b-474a-42c1-a576-2e0005a2a8c6/volumes"
Nov 02 18:13:34.074360 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.074316    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"
Nov 02 18:13:34.074360 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.074340    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt" event={"ID":"7e85b538-1702-463b-adc2-23276a23367e","Type":"ContainerDied","Data":"4a19ff00dd1f78f27d3d9c8b5bcae664021510ee5806de74371bf3dd649d8e48"}
Nov 02 18:13:34.074567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.074398    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="4a19ff00dd1f78f27d3d9c8b5bcae664021510ee5806de74371bf3dd649d8e48"
Nov 02 18:13:34.075939 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.075900    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"
Nov 02 18:13:34.076057 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.075951    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn" event={"ID":"d95809ee-3d8e-4197-9792-342b1c7a4820","Type":"ContainerDied","Data":"244bd0074db8ff678b031076d0db8e99a228c95dad6f1a6c9115504e11677d39"}
Nov 02 18:13:34.076057 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.076007    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="244bd0074db8ff678b031076d0db8e99a228c95dad6f1a6c9115504e11677d39"
Nov 02 18:13:34.077591 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.077570    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"
Nov 02 18:13:34.077701 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.077577    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp" event={"ID":"6cc67f3e-e8e3-4d57-859c-abb8561601a0","Type":"ContainerDied","Data":"bbf762b38cbe1d918c8eee9af0cc71c3c194c37bb4b7a7ed2e5e46bd6adf9f0a"}
Nov 02 18:13:34.077701 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.077694    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="bbf762b38cbe1d918c8eee9af0cc71c3c194c37bb4b7a7ed2e5e46bd6adf9f0a"
Nov 02 18:13:34.079159 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.079138    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc" event={"ID":"1d81d958-89bb-4f35-8c6c-d7a09d2507ae","Type":"ContainerDied","Data":"3ba6f59f2fd9e1d1e75d84ce16b58159389fa1f9872b11e0e91a09ed3a39497d"}
Nov 02 18:13:34.079249 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.079176    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="3ba6f59f2fd9e1d1e75d84ce16b58159389fa1f9872b11e0e91a09ed3a39497d"
Nov 02 18:13:34.079249 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.079227    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"
Nov 02 18:13:34.346759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.346668    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main" containerID="cri-o://da563580a735e237934140643b5625df59154f5b6d1ff73d1b0af154dc8aaf39" gracePeriod=45
Nov 02 18:13:34.365166 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.365134    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main" containerID="cri-o://20ff7d9e053f3451dc21448f1c411ba7fec1a711af2ca0d9541826d17ea0d8cb" gracePeriod=45
Nov 02 18:13:34.422737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.422713    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:34.652094 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.652064    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 18:13:34.672472 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.672434    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:13:34.763439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.763403    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-model-cache\") pod \"bffc7364-3157-46e6-8fba-616f128b60cf\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") "
Nov 02 18:13:34.763630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.763450    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-tmp-dir\") pod \"bffc7364-3157-46e6-8fba-616f128b60cf\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") "
Nov 02 18:13:34.763630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.763520    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-dshm\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-dshm\") pod \"bffc7364-3157-46e6-8fba-616f128b60cf\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") "
Nov 02 18:13:34.763761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.763647    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-home\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-home\") pod \"bffc7364-3157-46e6-8fba-616f128b60cf\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") "
Nov 02 18:13:34.763761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.763681    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/76d3e197-83a0-4793-8593-0b47b55a9fc4-kube-api-access-86tf4\" (UniqueName: \"kubernetes.io/projected/76d3e197-83a0-4793-8593-0b47b55a9fc4-kube-api-access-86tf4\") pod \"76d3e197-83a0-4793-8593-0b47b55a9fc4\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") "
Nov 02 18:13:34.763761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.763720    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-model-cache" pod "bffc7364-3157-46e6-8fba-616f128b60cf" (UID: "bffc7364-3157-46e6-8fba-616f128b60cf"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:34.763761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.763729    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-tmp-dir\") pod \"76d3e197-83a0-4793-8593-0b47b55a9fc4\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") "
Nov 02 18:13:34.763761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.763760    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-home\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-home\") pod \"76d3e197-83a0-4793-8593-0b47b55a9fc4\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") "
Nov 02 18:13:34.764097 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.763782    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/bffc7364-3157-46e6-8fba-616f128b60cf-kube-api-access-cbjks\" (UniqueName: \"kubernetes.io/projected/bffc7364-3157-46e6-8fba-616f128b60cf-kube-api-access-cbjks\") pod \"bffc7364-3157-46e6-8fba-616f128b60cf\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") "
Nov 02 18:13:34.764097 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.763827    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/bffc7364-3157-46e6-8fba-616f128b60cf-tls-certs\" (UniqueName: \"kubernetes.io/secret/bffc7364-3157-46e6-8fba-616f128b60cf-tls-certs\") pod \"bffc7364-3157-46e6-8fba-616f128b60cf\" (UID: \"bffc7364-3157-46e6-8fba-616f128b60cf\") "
Nov 02 18:13:34.764097 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.763857    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/76d3e197-83a0-4793-8593-0b47b55a9fc4-tls-certs\" (UniqueName: \"kubernetes.io/secret/76d3e197-83a0-4793-8593-0b47b55a9fc4-tls-certs\") pod \"76d3e197-83a0-4793-8593-0b47b55a9fc4\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") "
Nov 02 18:13:34.764097 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.763884    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-dshm\") pod \"76d3e197-83a0-4793-8593-0b47b55a9fc4\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") "
Nov 02 18:13:34.764097 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.763910    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-model-cache\") pod \"76d3e197-83a0-4793-8593-0b47b55a9fc4\" (UID: \"76d3e197-83a0-4793-8593-0b47b55a9fc4\") "
Nov 02 18:13:34.764394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.764073    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-home" pod "76d3e197-83a0-4793-8593-0b47b55a9fc4" (UID: "76d3e197-83a0-4793-8593-0b47b55a9fc4"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:34.764394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.764117    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-tmp-dir" pod "76d3e197-83a0-4793-8593-0b47b55a9fc4" (UID: "76d3e197-83a0-4793-8593-0b47b55a9fc4"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:34.764514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.764404    4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-model-cache\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:34.764514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.764422    4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-tmp-dir\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:34.764514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.764435    4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-home\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:34.765208 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.765177    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-tmp-dir" pod "bffc7364-3157-46e6-8fba-616f128b60cf" (UID: "bffc7364-3157-46e6-8fba-616f128b60cf"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:34.765614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.765305    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-home" pod "bffc7364-3157-46e6-8fba-616f128b60cf" (UID: "bffc7364-3157-46e6-8fba-616f128b60cf"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:34.765614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.765550    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-model-cache" pod "76d3e197-83a0-4793-8593-0b47b55a9fc4" (UID: "76d3e197-83a0-4793-8593-0b47b55a9fc4"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:34.767895 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.767424    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/76d3e197-83a0-4793-8593-0b47b55a9fc4-tls-certs" pod "76d3e197-83a0-4793-8593-0b47b55a9fc4" (UID: "76d3e197-83a0-4793-8593-0b47b55a9fc4"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:34.767895 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.767624    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/bffc7364-3157-46e6-8fba-616f128b60cf-tls-certs" pod "bffc7364-3157-46e6-8fba-616f128b60cf" (UID: "bffc7364-3157-46e6-8fba-616f128b60cf"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:13:34.767895 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.767862    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-dshm" pod "76d3e197-83a0-4793-8593-0b47b55a9fc4" (UID: "76d3e197-83a0-4793-8593-0b47b55a9fc4"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:34.768132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.767978    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/bffc7364-3157-46e6-8fba-616f128b60cf-kube-api-access-cbjks" pod "bffc7364-3157-46e6-8fba-616f128b60cf" (UID: "bffc7364-3157-46e6-8fba-616f128b60cf"). InnerVolumeSpecName "kube-api-access-cbjks". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:34.769187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.769162    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-dshm" pod "bffc7364-3157-46e6-8fba-616f128b60cf" (UID: "bffc7364-3157-46e6-8fba-616f128b60cf"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:34.769467 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.769446    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/76d3e197-83a0-4793-8593-0b47b55a9fc4-kube-api-access-86tf4" pod "76d3e197-83a0-4793-8593-0b47b55a9fc4" (UID: "76d3e197-83a0-4793-8593-0b47b55a9fc4"). InnerVolumeSpecName "kube-api-access-86tf4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:34.865279 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.865239    4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-dshm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:34.865279 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.865279    4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-home\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:34.865540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.865295    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-86tf4\" (UniqueName: \"kubernetes.io/projected/76d3e197-83a0-4793-8593-0b47b55a9fc4-kube-api-access-86tf4\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:34.865540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.865313    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-cbjks\" (UniqueName: \"kubernetes.io/projected/bffc7364-3157-46e6-8fba-616f128b60cf-kube-api-access-cbjks\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:34.865540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.865327    4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/bffc7364-3157-46e6-8fba-616f128b60cf-tls-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:34.865540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.865340    4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/76d3e197-83a0-4793-8593-0b47b55a9fc4-tls-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:34.865540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.865352    4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-dshm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:34.865540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.865365    4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/76d3e197-83a0-4793-8593-0b47b55a9fc4-model-cache\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:34.865540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:34.865377    4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/bffc7364-3157-46e6-8fba-616f128b60cf-tmp-dir\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:35.010902 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.010870    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:35.088488 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.088450    4456 generic.go:356] "Generic (PLEG): container finished" podID="bffc7364-3157-46e6-8fba-616f128b60cf" containerID="da563580a735e237934140643b5625df59154f5b6d1ff73d1b0af154dc8aaf39" exitCode=0
Nov 02 18:13:35.088646 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.088496    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"
Nov 02 18:13:35.088646 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.088537    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" event={"ID":"bffc7364-3157-46e6-8fba-616f128b60cf","Type":"ContainerDied","Data":"da563580a735e237934140643b5625df59154f5b6d1ff73d1b0af154dc8aaf39"}
Nov 02 18:13:35.088646 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.088583    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f" event={"ID":"bffc7364-3157-46e6-8fba-616f128b60cf","Type":"ContainerDied","Data":"b2484ce3e10e3880f991bd897ea3087a31547eab5fd9615eaa7df12023f7917b"}
Nov 02 18:13:35.088646 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.088602    4456 scope.go:122] "RemoveContainer" containerID="da563580a735e237934140643b5625df59154f5b6d1ff73d1b0af154dc8aaf39"
Nov 02 18:13:35.090278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.090241    4456 generic.go:356] "Generic (PLEG): container finished" podID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerID="20ff7d9e053f3451dc21448f1c411ba7fec1a711af2ca0d9541826d17ea0d8cb" exitCode=0
Nov 02 18:13:35.090383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.090286    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"
Nov 02 18:13:35.090383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.090303    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" event={"ID":"76d3e197-83a0-4793-8593-0b47b55a9fc4","Type":"ContainerDied","Data":"20ff7d9e053f3451dc21448f1c411ba7fec1a711af2ca0d9541826d17ea0d8cb"}
Nov 02 18:13:35.090383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.090333    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv" event={"ID":"76d3e197-83a0-4793-8593-0b47b55a9fc4","Type":"ContainerDied","Data":"8dc93a5b3e6f1b88f4d492ce6335a49a9df439e42177e1dfe5ffc1e7ea518989"}
Nov 02 18:13:35.102419 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.102395    4456 scope.go:122] "RemoveContainer" containerID="da563580a735e237934140643b5625df59154f5b6d1ff73d1b0af154dc8aaf39"
Nov 02 18:13:35.107204 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:35.107177    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"da563580a735e237934140643b5625df59154f5b6d1ff73d1b0af154dc8aaf39\": container with ID starting with da563580a735e237934140643b5625df59154f5b6d1ff73d1b0af154dc8aaf39 not found: ID does not exist" containerID="da563580a735e237934140643b5625df59154f5b6d1ff73d1b0af154dc8aaf39"
Nov 02 18:13:35.107320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.107216    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"da563580a735e237934140643b5625df59154f5b6d1ff73d1b0af154dc8aaf39"} err="failed to get container status \"da563580a735e237934140643b5625df59154f5b6d1ff73d1b0af154dc8aaf39\": rpc error: code = NotFound desc = could not find container \"da563580a735e237934140643b5625df59154f5b6d1ff73d1b0af154dc8aaf39\": container with ID starting with da563580a735e237934140643b5625df59154f5b6d1ff73d1b0af154dc8aaf39 not found: ID does not exist"
Nov 02 18:13:35.107320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.107240    4456 scope.go:122] "RemoveContainer" containerID="20ff7d9e053f3451dc21448f1c411ba7fec1a711af2ca0d9541826d17ea0d8cb"
Nov 02 18:13:35.117092 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.117065    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"]
Nov 02 18:13:35.121142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.121114    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w3-5eedab-4fdj7"]
Nov 02 18:13:35.121425 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.121401    4456 scope.go:122] "RemoveContainer" containerID="20ff7d9e053f3451dc21448f1c411ba7fec1a711af2ca0d9541826d17ea0d8cb"
Nov 02 18:13:35.121751 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:35.121728    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"20ff7d9e053f3451dc21448f1c411ba7fec1a711af2ca0d9541826d17ea0d8cb\": container with ID starting with 20ff7d9e053f3451dc21448f1c411ba7fec1a711af2ca0d9541826d17ea0d8cb not found: ID does not exist" containerID="20ff7d9e053f3451dc21448f1c411ba7fec1a711af2ca0d9541826d17ea0d8cb"
Nov 02 18:13:35.121803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.121764    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"20ff7d9e053f3451dc21448f1c411ba7fec1a711af2ca0d9541826d17ea0d8cb"} err="failed to get container status \"20ff7d9e053f3451dc21448f1c411ba7fec1a711af2ca0d9541826d17ea0d8cb\": rpc error: code = NotFound desc = could not find container \"20ff7d9e053f3451dc21448f1c411ba7fec1a711af2ca0d9541826d17ea0d8cb\": container with ID starting with 20ff7d9e053f3451dc21448f1c411ba7fec1a711af2ca0d9541826d17ea0d8cb not found: ID does not exist"
Nov 02 18:13:35.150598 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.150554    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:35.161572 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.161540    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"]
Nov 02 18:13:35.166841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.166797    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/distinct-2-w5-863b87-kserve-f8c7cd565-kmv7f"]
Nov 02 18:13:35.202441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.202416    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"]
Nov 02 18:13:35.207764 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.207743    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w5-863b87/simulator-w5-863b87-kserve-bd54699cd-7g2mv"]
Nov 02 18:13:35.596647 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.596615    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:35.612955 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.612929    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="327124e1-a104-41ad-805c-1eb5363445fa" path="/var/lib/kubelet/pods/327124e1-a104-41ad-805c-1eb5363445fa/volumes"
Nov 02 18:13:35.613336 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.613323    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" path="/var/lib/kubelet/pods/76d3e197-83a0-4793-8593-0b47b55a9fc4/volumes"
Nov 02 18:13:35.613683 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:35.613671    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" path="/var/lib/kubelet/pods/bffc7364-3157-46e6-8fba-616f128b60cf/volumes"
Nov 02 18:13:36.694926 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:36.694891    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:37.085034 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:37.084935    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:38.161752 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:38.161714    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:38.313095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:38.313059    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:38.960497 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:38.960466    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:39.397273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:39.397176    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:39.541792 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:39.541761    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:39.774199 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:39.774163    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:40.761487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:40.761454    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:42.635327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:42.635272    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:43.049720 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:43.048580    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:43.200945 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:43.200910    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:43.331101 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:43.331017    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"]
Nov 02 18:13:43.331324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:43.331288    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy" containerID="cri-o://4db8580f4eb63d66c47f9f317c68868ee1b4d374974981f9e830b2a3f1cea2f5" gracePeriod=30
Nov 02 18:13:43.392073 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:43.392032    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:44.460874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:44.460798    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:46.166242 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:46.166199    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:13:46.326777 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:46.326743    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:48.145691 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.145333    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:48.618189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.618163    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 18:13:48.623512 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.623484    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-envoy\") pod \"f56cfa88-cc79-4d08-9111-b47d940533ba\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") "
Nov 02 18:13:48.623622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.623548    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-podinfo\") pod \"f56cfa88-cc79-4d08-9111-b47d940533ba\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") "
Nov 02 18:13:48.623622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.623583    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-token\" (UniqueName: \"kubernetes.io/projected/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-token\") pod \"f56cfa88-cc79-4d08-9111-b47d940533ba\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") "
Nov 02 18:13:48.623750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.623619    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f56cfa88-cc79-4d08-9111-b47d940533ba-kube-api-access-zq8ts\" (UniqueName: \"kubernetes.io/projected/f56cfa88-cc79-4d08-9111-b47d940533ba-kube-api-access-zq8ts\") pod \"f56cfa88-cc79-4d08-9111-b47d940533ba\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") "
Nov 02 18:13:48.623750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.623678    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-workload-socket\") pod \"f56cfa88-cc79-4d08-9111-b47d940533ba\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") "
Nov 02 18:13:48.623750 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.623708    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f56cfa88-cc79-4d08-9111-b47d940533ba-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f56cfa88-cc79-4d08-9111-b47d940533ba-istiod-ca-cert\") pod \"f56cfa88-cc79-4d08-9111-b47d940533ba\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") "
Nov 02 18:13:48.623915 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.623784    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-data\") pod \"f56cfa88-cc79-4d08-9111-b47d940533ba\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") "
Nov 02 18:13:48.623915 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.623866    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-credential-socket\") pod \"f56cfa88-cc79-4d08-9111-b47d940533ba\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") "
Nov 02 18:13:48.623915 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.623892    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-workload-certs\") pod \"f56cfa88-cc79-4d08-9111-b47d940533ba\" (UID: \"f56cfa88-cc79-4d08-9111-b47d940533ba\") "
Nov 02 18:13:48.624068 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.624040    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-workload-socket" pod "f56cfa88-cc79-4d08-9111-b47d940533ba" (UID: "f56cfa88-cc79-4d08-9111-b47d940533ba"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:48.624925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.624249    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:48.624925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.624358    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-credential-socket" pod "f56cfa88-cc79-4d08-9111-b47d940533ba" (UID: "f56cfa88-cc79-4d08-9111-b47d940533ba"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:48.624925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.624777    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f56cfa88-cc79-4d08-9111-b47d940533ba-istiod-ca-cert" pod "f56cfa88-cc79-4d08-9111-b47d940533ba" (UID: "f56cfa88-cc79-4d08-9111-b47d940533ba"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:13:48.624925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.624804    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-data" pod "f56cfa88-cc79-4d08-9111-b47d940533ba" (UID: "f56cfa88-cc79-4d08-9111-b47d940533ba"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:48.624925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.624879    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-workload-certs" pod "f56cfa88-cc79-4d08-9111-b47d940533ba" (UID: "f56cfa88-cc79-4d08-9111-b47d940533ba"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:48.626268 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.626232    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-envoy" pod "f56cfa88-cc79-4d08-9111-b47d940533ba" (UID: "f56cfa88-cc79-4d08-9111-b47d940533ba"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:13:48.626395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.626270    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-token" pod "f56cfa88-cc79-4d08-9111-b47d940533ba" (UID: "f56cfa88-cc79-4d08-9111-b47d940533ba"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:48.626603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.626583    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-podinfo" pod "f56cfa88-cc79-4d08-9111-b47d940533ba" (UID: "f56cfa88-cc79-4d08-9111-b47d940533ba"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:13:48.626716 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.626695    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f56cfa88-cc79-4d08-9111-b47d940533ba-kube-api-access-zq8ts" pod "f56cfa88-cc79-4d08-9111-b47d940533ba" (UID: "f56cfa88-cc79-4d08-9111-b47d940533ba"). InnerVolumeSpecName "kube-api-access-zq8ts". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:13:48.725517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.725420    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:48.725517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.725452    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zq8ts\" (UniqueName: \"kubernetes.io/projected/f56cfa88-cc79-4d08-9111-b47d940533ba-kube-api-access-zq8ts\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:48.725517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.725461    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/f56cfa88-cc79-4d08-9111-b47d940533ba-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:48.725517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.725470    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:48.725517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.725478    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:48.725517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.725485    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:48.725517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.725493    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:48.725517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:48.725501    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/f56cfa88-cc79-4d08-9111-b47d940533ba-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:13:49.199926 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:49.199893    4456 generic.go:356] "Generic (PLEG): container finished" podID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerID="4db8580f4eb63d66c47f9f317c68868ee1b4d374974981f9e830b2a3f1cea2f5" exitCode=0
Nov 02 18:13:49.200402 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:49.199968    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"
Nov 02 18:13:49.200402 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:49.199993    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" event={"ID":"f56cfa88-cc79-4d08-9111-b47d940533ba","Type":"ContainerDied","Data":"4db8580f4eb63d66c47f9f317c68868ee1b4d374974981f9e830b2a3f1cea2f5"}
Nov 02 18:13:49.200402 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:49.200054    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7" event={"ID":"f56cfa88-cc79-4d08-9111-b47d940533ba","Type":"ContainerDied","Data":"22c6aba7e42ad31aa390d0dac09323af68b95770b49f8b3199b2570582875c02"}
Nov 02 18:13:49.200402 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:49.200082    4456 scope.go:122] "RemoveContainer" containerID="4db8580f4eb63d66c47f9f317c68868ee1b4d374974981f9e830b2a3f1cea2f5"
Nov 02 18:13:49.214753 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:49.214732    4456 scope.go:122] "RemoveContainer" containerID="4db8580f4eb63d66c47f9f317c68868ee1b4d374974981f9e830b2a3f1cea2f5"
Nov 02 18:13:49.215064 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:13:49.215039    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4db8580f4eb63d66c47f9f317c68868ee1b4d374974981f9e830b2a3f1cea2f5\": container with ID starting with 4db8580f4eb63d66c47f9f317c68868ee1b4d374974981f9e830b2a3f1cea2f5 not found: ID does not exist" containerID="4db8580f4eb63d66c47f9f317c68868ee1b4d374974981f9e830b2a3f1cea2f5"
Nov 02 18:13:49.215135 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:49.215076    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4db8580f4eb63d66c47f9f317c68868ee1b4d374974981f9e830b2a3f1cea2f5"} err="failed to get container status \"4db8580f4eb63d66c47f9f317c68868ee1b4d374974981f9e830b2a3f1cea2f5\": rpc error: code = NotFound desc = could not find container \"4db8580f4eb63d66c47f9f317c68868ee1b4d374974981f9e830b2a3f1cea2f5\": container with ID starting with 4db8580f4eb63d66c47f9f317c68868ee1b4d374974981f9e830b2a3f1cea2f5 not found: ID does not exist"
Nov 02 18:13:49.273664 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:49.273630    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"]
Nov 02 18:13:49.287725 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:49.287686    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-w3-5eedab-openshift-default-678f4dc9f9-p42p7"]
Nov 02 18:13:49.614692 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:49.614611    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" path="/var/lib/kubelet/pods/f56cfa88-cc79-4d08-9111-b47d940533ba/volumes"
Nov 02 18:13:49.727365 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:49.727329    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:49.942461 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:49.942432    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:50.087737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:50.087704    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:50.630257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:50.630217    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:50.855917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:50.855862    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:51.468078 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:51.468045    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:51.611879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:51.611840    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:51.909324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:51.909242    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:52.113946 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:52.113906    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:52.195384 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:52.195356    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"]
Nov 02 18:13:52.208043 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:52.208014    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w4-2899ab-wh4xj"]
Nov 02 18:13:52.276033 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:52.275997    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:52.645319 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:52.645241    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:52.771031 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:52.771001    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"]
Nov 02 18:13:52.783999 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:52.783950    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w5-863b87-zrf5p"]
Nov 02 18:13:52.786770 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:52.786716    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:52.924757 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:52.924669    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:53.293720 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:53.293680    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:53.615135 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:53.615057    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="116e4234-31db-4d7f-b00d-e418850d25b6" path="/var/lib/kubelet/pods/116e4234-31db-4d7f-b00d-e418850d25b6/volumes"
Nov 02 18:13:53.615651 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:53.615634    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="be3edfab-727f-442d-8fe3-176358694c84" path="/var/lib/kubelet/pods/be3edfab-727f-442d-8fe3-176358694c84/volumes"
Nov 02 18:13:53.900089 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:53.899994    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:54.043834 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:54.043789    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:54.211860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:54.211793    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:54.674768 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:54.674679    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:55.050099 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:55.050067    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:55.197469 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:55.197419    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:56.305288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:56.305227    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:56.457664 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:56.457593    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:57.809593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:57.809562    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:58.067448 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:58.067364    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:58.356378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:58.356249    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:59.049102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:59.049023    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:13:59.193982 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:13:59.193931    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:00.001333 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:00.001296    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:00.163490 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:00.163432    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:00.317913 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:00.317826    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:00.461297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:00.461259    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:00.847301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:00.847249    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:01.009683 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:01.009646    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:01.203073 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:01.203034    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:02.206677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:02.206641    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:03.691571 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:03.691521    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:03.947949 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:03.947790    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:05.332301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:05.332244    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:05.651305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:05.651202    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:07.365432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:07.365399    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:09.168951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:09.168901    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:11.050939 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:11.050906    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:12.259832 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:12.259788    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:12.856627 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:12.856586    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:14.654127 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:14.654088    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:16.166635 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:14:16.166594    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:14:16.486507 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:16.486467    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:18.356161 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:18.356121    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:20.167914 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:20.167882    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:21.949509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:21.949476    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:23.813980 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:23.813945    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:25.469726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:25.469690    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:27.254282 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:27.254251    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:28.954023 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:28.953984    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:29.907538 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.907503    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"]
Nov 02 18:14:29.907969 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.907956    4456 cpu_manager.go:414] "containerMap: removing container" podUID="7e85b538-1702-463b-adc2-23276a23367e" containerName="revoke-keys"
Nov 02 18:14:29.908042 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.907971    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7e85b538-1702-463b-adc2-23276a23367e" containerName="revoke-keys"
Nov 02 18:14:29.908042 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.907988    4456 cpu_manager.go:414] "containerMap: removing container" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main"
Nov 02 18:14:29.908042 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.907994    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main"
Nov 02 18:14:29.908042 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908009    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d95809ee-3d8e-4197-9792-342b1c7a4820" containerName="revoke-keys"
Nov 02 18:14:29.908042 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908015    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d95809ee-3d8e-4197-9792-342b1c7a4820" containerName="revoke-keys"
Nov 02 18:14:29.908042 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908026    4456 cpu_manager.go:414] "containerMap: removing container" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main"
Nov 02 18:14:29.908042 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908030    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main"
Nov 02 18:14:29.908042 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908042    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy"
Nov 02 18:14:29.908042 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908047    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy"
Nov 02 18:14:29.908335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908066    4456 cpu_manager.go:414] "containerMap: removing container" podUID="1d81d958-89bb-4f35-8c6c-d7a09d2507ae" containerName="revoke-keys"
Nov 02 18:14:29.908335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908070    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="1d81d958-89bb-4f35-8c6c-d7a09d2507ae" containerName="revoke-keys"
Nov 02 18:14:29.908335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908082    4456 cpu_manager.go:414] "containerMap: removing container" podUID="6cc67f3e-e8e3-4d57-859c-abb8561601a0" containerName="revoke-keys"
Nov 02 18:14:29.908335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908090    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6cc67f3e-e8e3-4d57-859c-abb8561601a0" containerName="revoke-keys"
Nov 02 18:14:29.908335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908174    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f56cfa88-cc79-4d08-9111-b47d940533ba" containerName="istio-proxy"
Nov 02 18:14:29.908335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908190    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="1d81d958-89bb-4f35-8c6c-d7a09d2507ae" containerName="revoke-keys"
Nov 02 18:14:29.908335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908202    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6cc67f3e-e8e3-4d57-859c-abb8561601a0" containerName="revoke-keys"
Nov 02 18:14:29.908335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908209    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7e85b538-1702-463b-adc2-23276a23367e" containerName="revoke-keys"
Nov 02 18:14:29.908335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908221    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="bffc7364-3157-46e6-8fba-616f128b60cf" containerName="main"
Nov 02 18:14:29.908335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908232    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d95809ee-3d8e-4197-9792-342b1c7a4820" containerName="revoke-keys"
Nov 02 18:14:29.908335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.908240    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="76d3e197-83a0-4793-8593-0b47b55a9fc4" containerName="main"
Nov 02 18:14:29.914308 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.914292    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"
Nov 02 18:14:29.944540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.944514    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10825c71-87c7-4b3d-b09a-f17130b1b391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bebed52-74hfl\" (UID: \"10825c71-87c7-4b3d-b09a-f17130b1b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"
Nov 02 18:14:29.944653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.944552    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10825c71-87c7-4b3d-b09a-f17130b1b391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bebed52-74hfl\" (UID: \"10825c71-87c7-4b3d-b09a-f17130b1b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"
Nov 02 18:14:29.944695 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.944686    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rsgk5\" (UniqueName: \"kubernetes.io/projected/10825c71-87c7-4b3d-b09a-f17130b1b391-kube-api-access-rsgk5\") pod \"maas-api-revoke-sub-0bebed52-74hfl\" (UID: \"10825c71-87c7-4b3d-b09a-f17130b1b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"
Nov 02 18:14:29.954187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.954161    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:14:29.954595 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.954313    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:14:29.954595 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.954367    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:14:29.954595 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.954400    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:14:29.957492 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:29.957470    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"]
Nov 02 18:14:30.045425 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:30.045393    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10825c71-87c7-4b3d-b09a-f17130b1b391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bebed52-74hfl\" (UID: \"10825c71-87c7-4b3d-b09a-f17130b1b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"
Nov 02 18:14:30.045554 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:30.045504    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rsgk5\" (UniqueName: \"kubernetes.io/projected/10825c71-87c7-4b3d-b09a-f17130b1b391-kube-api-access-rsgk5\") pod \"maas-api-revoke-sub-0bebed52-74hfl\" (UID: \"10825c71-87c7-4b3d-b09a-f17130b1b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"
Nov 02 18:14:30.045620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:30.045606    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10825c71-87c7-4b3d-b09a-f17130b1b391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bebed52-74hfl\" (UID: \"10825c71-87c7-4b3d-b09a-f17130b1b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"
Nov 02 18:14:30.046322 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:30.046247    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10825c71-87c7-4b3d-b09a-f17130b1b391-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0bebed52-74hfl\" (UID: \"10825c71-87c7-4b3d-b09a-f17130b1b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"
Nov 02 18:14:30.048010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:30.047989    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10825c71-87c7-4b3d-b09a-f17130b1b391-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0bebed52-74hfl\" (UID: \"10825c71-87c7-4b3d-b09a-f17130b1b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"
Nov 02 18:14:30.069572 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:30.069548    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rsgk5\" (UniqueName: \"kubernetes.io/projected/10825c71-87c7-4b3d-b09a-f17130b1b391-kube-api-access-rsgk5\") pod \"maas-api-revoke-sub-0bebed52-74hfl\" (UID: \"10825c71-87c7-4b3d-b09a-f17130b1b391\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"
Nov 02 18:14:30.227652 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:30.227601    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"
Nov 02 18:14:30.525612 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:30.525583    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"]
Nov 02 18:14:30.527170 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:14:30.527143    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod10825c71_87c7_4b3d_b09a_f17130b1b391.slice/crio-6375862d1f55f526f924b2f433c5a70becf7f6b0ad127084c6e1a2cf721db850 WatchSource:0}: Error finding container 6375862d1f55f526f924b2f433c5a70becf7f6b0ad127084c6e1a2cf721db850: Status 404 returned error can't find the container with id 6375862d1f55f526f924b2f433c5a70becf7f6b0ad127084c6e1a2cf721db850
Nov 02 18:14:30.656608 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:30.656566    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:31.495770 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:31.495731    4456 generic.go:356] "Generic (PLEG): container finished" podID="10825c71-87c7-4b3d-b09a-f17130b1b391" containerID="d1a453c9869097ca10c33155469b8043bbc8a267124a052cd5724b3e0dcee129" exitCode=0
Nov 02 18:14:31.495770 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:31.495767    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl" event={"ID":"10825c71-87c7-4b3d-b09a-f17130b1b391","Type":"ContainerDied","Data":"d1a453c9869097ca10c33155469b8043bbc8a267124a052cd5724b3e0dcee129"}
Nov 02 18:14:31.496269 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:31.495876    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl" event={"ID":"10825c71-87c7-4b3d-b09a-f17130b1b391","Type":"ContainerStarted","Data":"6375862d1f55f526f924b2f433c5a70becf7f6b0ad127084c6e1a2cf721db850"}
Nov 02 18:14:32.355438 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:32.355402    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:32.657637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:32.657611    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"
Nov 02 18:14:32.777793 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:32.777766    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/10825c71-87c7-4b3d-b09a-f17130b1b391-kube-api-access-rsgk5\" (UniqueName: \"kubernetes.io/projected/10825c71-87c7-4b3d-b09a-f17130b1b391-kube-api-access-rsgk5\") pod \"10825c71-87c7-4b3d-b09a-f17130b1b391\" (UID: \"10825c71-87c7-4b3d-b09a-f17130b1b391\") "
Nov 02 18:14:32.777953 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:32.777806    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/10825c71-87c7-4b3d-b09a-f17130b1b391-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10825c71-87c7-4b3d-b09a-f17130b1b391-maas-api-xks-ca\") pod \"10825c71-87c7-4b3d-b09a-f17130b1b391\" (UID: \"10825c71-87c7-4b3d-b09a-f17130b1b391\") "
Nov 02 18:14:32.777953 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:32.777905    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/10825c71-87c7-4b3d-b09a-f17130b1b391-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10825c71-87c7-4b3d-b09a-f17130b1b391-maas-api-openshift-ca\") pod \"10825c71-87c7-4b3d-b09a-f17130b1b391\" (UID: \"10825c71-87c7-4b3d-b09a-f17130b1b391\") "
Nov 02 18:14:32.778265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:32.778237    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/10825c71-87c7-4b3d-b09a-f17130b1b391-maas-api-openshift-ca" pod "10825c71-87c7-4b3d-b09a-f17130b1b391" (UID: "10825c71-87c7-4b3d-b09a-f17130b1b391"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:14:32.780274 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:32.780252    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/10825c71-87c7-4b3d-b09a-f17130b1b391-kube-api-access-rsgk5" pod "10825c71-87c7-4b3d-b09a-f17130b1b391" (UID: "10825c71-87c7-4b3d-b09a-f17130b1b391"). InnerVolumeSpecName "kube-api-access-rsgk5". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:14:32.780703 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:32.780675    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/10825c71-87c7-4b3d-b09a-f17130b1b391-maas-api-xks-ca" pod "10825c71-87c7-4b3d-b09a-f17130b1b391" (UID: "10825c71-87c7-4b3d-b09a-f17130b1b391"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:14:32.879609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:32.879579    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/10825c71-87c7-4b3d-b09a-f17130b1b391-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:14:32.879609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:32.879607    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rsgk5\" (UniqueName: \"kubernetes.io/projected/10825c71-87c7-4b3d-b09a-f17130b1b391-kube-api-access-rsgk5\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:14:32.879752 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:32.879618    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/10825c71-87c7-4b3d-b09a-f17130b1b391-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:14:33.514527 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:33.514493    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl" event={"ID":"10825c71-87c7-4b3d-b09a-f17130b1b391","Type":"ContainerDied","Data":"6375862d1f55f526f924b2f433c5a70becf7f6b0ad127084c6e1a2cf721db850"}
Nov 02 18:14:33.514724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:33.514537    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"
Nov 02 18:14:33.514724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:33.514551    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6375862d1f55f526f924b2f433c5a70becf7f6b0ad127084c6e1a2cf721db850"
Nov 02 18:14:34.158898 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:34.158862    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:35.859913 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:35.859869    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:37.602866 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:37.602832    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:38.271674 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:38.271643    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:39.356785 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:39.356752    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:39.923302 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:39.923272    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:40.322648 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.322612    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"]
Nov 02 18:14:40.323072 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.323058    4456 cpu_manager.go:414] "containerMap: removing container" podUID="10825c71-87c7-4b3d-b09a-f17130b1b391" containerName="revoke-keys"
Nov 02 18:14:40.323121 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.323073    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="10825c71-87c7-4b3d-b09a-f17130b1b391" containerName="revoke-keys"
Nov 02 18:14:40.323160 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.323146    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="10825c71-87c7-4b3d-b09a-f17130b1b391" containerName="revoke-keys"
Nov 02 18:14:40.325906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.325887    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"
Nov 02 18:14:40.348102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.348080    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"]
Nov 02 18:14:40.357398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.357376    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:14:40.357398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.357390    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:14:40.357754 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.357429    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:14:40.357754 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.357464    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:14:40.456705 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.456673    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46e4abcd-4dz8j\" (UID: \"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"
Nov 02 18:14:40.456943 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.456727    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tx6dm\" (UniqueName: \"kubernetes.io/projected/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-kube-api-access-tx6dm\") pod \"maas-api-revoke-sub-46e4abcd-4dz8j\" (UID: \"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"
Nov 02 18:14:40.456943 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.456871    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46e4abcd-4dz8j\" (UID: \"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"
Nov 02 18:14:40.558316 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.558283    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46e4abcd-4dz8j\" (UID: \"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"
Nov 02 18:14:40.558507 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.558337    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tx6dm\" (UniqueName: \"kubernetes.io/projected/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-kube-api-access-tx6dm\") pod \"maas-api-revoke-sub-46e4abcd-4dz8j\" (UID: \"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"
Nov 02 18:14:40.558507 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.558391    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46e4abcd-4dz8j\" (UID: \"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"
Nov 02 18:14:40.559006 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.558984    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-46e4abcd-4dz8j\" (UID: \"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"
Nov 02 18:14:40.560678 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.560658    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-46e4abcd-4dz8j\" (UID: \"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"
Nov 02 18:14:40.598177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.598113    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tx6dm\" (UniqueName: \"kubernetes.io/projected/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-kube-api-access-tx6dm\") pod \"maas-api-revoke-sub-46e4abcd-4dz8j\" (UID: \"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"
Nov 02 18:14:40.639221 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.639193    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"
Nov 02 18:14:40.937919 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:40.937895    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"]
Nov 02 18:14:40.939904 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:14:40.939880    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod32271bff_bd2b_43a8_8da1_cc7ae9b43cd7.slice/crio-8a071a16497769309a772d89cc44aebc623a53c52e56a5dd295a0ddad031207d WatchSource:0}: Error finding container 8a071a16497769309a772d89cc44aebc623a53c52e56a5dd295a0ddad031207d: Status 404 returned error can't find the container with id 8a071a16497769309a772d89cc44aebc623a53c52e56a5dd295a0ddad031207d
Nov 02 18:14:41.184525 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:41.184496    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:41.567931 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:41.567852    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j" event={"ID":"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7","Type":"ContainerStarted","Data":"7c8d5209e30f79dfe754b4d00ec10a49f5dd444755c89521e87700bbcc6163ca"}
Nov 02 18:14:41.567931 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:41.567918    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j" event={"ID":"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7","Type":"ContainerStarted","Data":"8a071a16497769309a772d89cc44aebc623a53c52e56a5dd295a0ddad031207d"}
Nov 02 18:14:41.613694 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:41.613645    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j" podStartSLOduration=1.613632266 podStartE2EDuration="1.613632266s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:40 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:41.610595495 +0000 UTC m=+2698.101658655" watchObservedRunningTime="2026-11-02 18:14:41.613632266 +0000 UTC m=+2698.104695475"
Nov 02 18:14:42.576930 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:42.576889    4456 generic.go:356] "Generic (PLEG): container finished" podID="32271bff-bd2b-43a8-8da1-cc7ae9b43cd7" containerID="7c8d5209e30f79dfe754b4d00ec10a49f5dd444755c89521e87700bbcc6163ca" exitCode=0
Nov 02 18:14:42.577311 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:42.576945    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j" event={"ID":"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7","Type":"ContainerDied","Data":"7c8d5209e30f79dfe754b4d00ec10a49f5dd444755c89521e87700bbcc6163ca"}
Nov 02 18:14:42.976718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:42.976671    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:43.732552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:43.732529    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"
Nov 02 18:14:43.794078 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:43.794045    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-kube-api-access-tx6dm\" (UniqueName: \"kubernetes.io/projected/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-kube-api-access-tx6dm\") pod \"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7\" (UID: \"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7\") "
Nov 02 18:14:43.794222 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:43.794181    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-maas-api-xks-ca\") pod \"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7\" (UID: \"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7\") "
Nov 02 18:14:43.794318 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:43.794301    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-maas-api-openshift-ca\") pod \"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7\" (UID: \"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7\") "
Nov 02 18:14:43.794680 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:43.794654    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-maas-api-openshift-ca" pod "32271bff-bd2b-43a8-8da1-cc7ae9b43cd7" (UID: "32271bff-bd2b-43a8-8da1-cc7ae9b43cd7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:14:43.796398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:43.796369    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-kube-api-access-tx6dm" pod "32271bff-bd2b-43a8-8da1-cc7ae9b43cd7" (UID: "32271bff-bd2b-43a8-8da1-cc7ae9b43cd7"). InnerVolumeSpecName "kube-api-access-tx6dm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:14:43.796490 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:43.796406    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-maas-api-xks-ca" pod "32271bff-bd2b-43a8-8da1-cc7ae9b43cd7" (UID: "32271bff-bd2b-43a8-8da1-cc7ae9b43cd7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:14:43.895440 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:43.895365    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-tx6dm\" (UniqueName: \"kubernetes.io/projected/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-kube-api-access-tx6dm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:14:43.895440 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:43.895393    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:14:43.895440 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:43.895403    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:14:44.592118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:44.592087    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"
Nov 02 18:14:44.592293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:44.592131    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j" event={"ID":"32271bff-bd2b-43a8-8da1-cc7ae9b43cd7","Type":"ContainerDied","Data":"8a071a16497769309a772d89cc44aebc623a53c52e56a5dd295a0ddad031207d"}
Nov 02 18:14:44.592293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:44.592227    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8a071a16497769309a772d89cc44aebc623a53c52e56a5dd295a0ddad031207d"
Nov 02 18:14:44.643262 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:44.643201    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:44.804144 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:44.804093    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:44.944585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:44.944550    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:46.164969 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:14:46.164932    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:14:46.447206 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:46.447172    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:48.240699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:48.240665    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:49.993272 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:49.993236    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:51.651020 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:51.650989    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:52.711320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:52.711281    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:53.442404 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:53.442371    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:54.334384 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.334341    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"]
Nov 02 18:14:54.419063 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.419019    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:54.617627 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.617545    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"]
Nov 02 18:14:54.618030 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.618014    4456 cpu_manager.go:414] "containerMap: removing container" podUID="32271bff-bd2b-43a8-8da1-cc7ae9b43cd7" containerName="revoke-keys"
Nov 02 18:14:54.618030 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.618031    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="32271bff-bd2b-43a8-8da1-cc7ae9b43cd7" containerName="revoke-keys"
Nov 02 18:14:54.618138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.618115    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="32271bff-bd2b-43a8-8da1-cc7ae9b43cd7" containerName="revoke-keys"
Nov 02 18:14:54.621533 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.621513    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"
Nov 02 18:14:54.637679 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.637652    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:14:54.637679 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.637658    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:14:54.638381 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.638353    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:14:54.638460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.638430    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:14:54.640929 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.640908    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"]
Nov 02 18:14:54.721266 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.721229    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn\" (UID: \"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"
Nov 02 18:14:54.721462 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.721283    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn\" (UID: \"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"
Nov 02 18:14:54.721462 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.721445    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d87z8\" (UniqueName: \"kubernetes.io/projected/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-kube-api-access-d87z8\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn\" (UID: \"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"
Nov 02 18:14:54.822761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.822723    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn\" (UID: \"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"
Nov 02 18:14:54.822940 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.822791    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn\" (UID: \"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"
Nov 02 18:14:54.822998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.822936    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d87z8\" (UniqueName: \"kubernetes.io/projected/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-kube-api-access-d87z8\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn\" (UID: \"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"
Nov 02 18:14:54.823658 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.823628    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn\" (UID: \"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"
Nov 02 18:14:54.825333 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.825311    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn\" (UID: \"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"
Nov 02 18:14:54.847776 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.847747    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d87z8\" (UniqueName: \"kubernetes.io/projected/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-kube-api-access-d87z8\") pod \"maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn\" (UID: \"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"
Nov 02 18:14:54.935439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:54.935408    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"
Nov 02 18:14:55.235980 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:55.235952    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"]
Nov 02 18:14:55.236443 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:14:55.236415    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9f1070de_7aa8_453e_b11e_17fc5fa3a1d5.slice/crio-272f7e7b981179693af40175d2f561fc137c36a43c4f89e60ef32e8130abc3bf WatchSource:0}: Error finding container 272f7e7b981179693af40175d2f561fc137c36a43c4f89e60ef32e8130abc3bf: Status 404 returned error can't find the container with id 272f7e7b981179693af40175d2f561fc137c36a43c4f89e60ef32e8130abc3bf
Nov 02 18:14:55.310454 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:55.310409    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:55.679886 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:55.679809    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn" event={"ID":"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5","Type":"ContainerStarted","Data":"d6b3fc0f71a003cf4eb46571b216bb2b835f695c67966d97a48bb40e5c80ba76"}
Nov 02 18:14:55.680478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:55.680442    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn" event={"ID":"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5","Type":"ContainerStarted","Data":"272f7e7b981179693af40175d2f561fc137c36a43c4f89e60ef32e8130abc3bf"}
Nov 02 18:14:55.723703 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:55.723654    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn" podStartSLOduration=1.723641426 podStartE2EDuration="1.723641426s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:14:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:14:55.723370399 +0000 UTC m=+2712.214433581" watchObservedRunningTime="2026-11-02 18:14:55.723641426 +0000 UTC m=+2712.214704586"
Nov 02 18:14:56.687826 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:56.687787    4456 generic.go:356] "Generic (PLEG): container finished" podID="9f1070de-7aa8-453e-b11e-17fc5fa3a1d5" containerID="d6b3fc0f71a003cf4eb46571b216bb2b835f695c67966d97a48bb40e5c80ba76" exitCode=0
Nov 02 18:14:56.688250 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:56.687884    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn" event={"ID":"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5","Type":"ContainerDied","Data":"d6b3fc0f71a003cf4eb46571b216bb2b835f695c67966d97a48bb40e5c80ba76"}
Nov 02 18:14:57.268093 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:57.268056    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:57.843860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:57.843836    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"
Nov 02 18:14:57.858145 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:57.858123    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-kube-api-access-d87z8\" (UniqueName: \"kubernetes.io/projected/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-kube-api-access-d87z8\") pod \"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5\" (UID: \"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5\") "
Nov 02 18:14:57.858246 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:57.858189    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-maas-api-xks-ca\") pod \"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5\" (UID: \"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5\") "
Nov 02 18:14:57.858246 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:57.858220    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-maas-api-openshift-ca\") pod \"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5\" (UID: \"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5\") "
Nov 02 18:14:57.858663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:57.858644    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-maas-api-openshift-ca" pod "9f1070de-7aa8-453e-b11e-17fc5fa3a1d5" (UID: "9f1070de-7aa8-453e-b11e-17fc5fa3a1d5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:14:57.860351 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:57.860326    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-kube-api-access-d87z8" pod "9f1070de-7aa8-453e-b11e-17fc5fa3a1d5" (UID: "9f1070de-7aa8-453e-b11e-17fc5fa3a1d5"). InnerVolumeSpecName "kube-api-access-d87z8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:14:57.860545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:57.860524    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-maas-api-xks-ca" pod "9f1070de-7aa8-453e-b11e-17fc5fa3a1d5" (UID: "9f1070de-7aa8-453e-b11e-17fc5fa3a1d5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:14:57.959356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:57.959278    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d87z8\" (UniqueName: \"kubernetes.io/projected/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-kube-api-access-d87z8\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:14:57.959356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:57.959310    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:14:57.959356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:57.959319    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:14:58.125802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:58.125765    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:14:58.701781 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:58.701747    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"
Nov 02 18:14:58.702009 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:58.701790    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn" event={"ID":"9f1070de-7aa8-453e-b11e-17fc5fa3a1d5","Type":"ContainerDied","Data":"272f7e7b981179693af40175d2f561fc137c36a43c4f89e60ef32e8130abc3bf"}
Nov 02 18:14:58.702009 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:58.701868    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="272f7e7b981179693af40175d2f561fc137c36a43c4f89e60ef32e8130abc3bf"
Nov 02 18:14:58.963370 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:14:58.963261    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:00.149398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.149366    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"]
Nov 02 18:15:00.149881 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.149809    4456 cpu_manager.go:414] "containerMap: removing container" podUID="9f1070de-7aa8-453e-b11e-17fc5fa3a1d5" containerName="revoke-keys"
Nov 02 18:15:00.149881 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.149846    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9f1070de-7aa8-453e-b11e-17fc5fa3a1d5" containerName="revoke-keys"
Nov 02 18:15:00.149959 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.149918    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="9f1070de-7aa8-453e-b11e-17fc5fa3a1d5" containerName="revoke-keys"
Nov 02 18:15:00.153524 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.153500    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"]
Nov 02 18:15:00.153645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.153630    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"
Nov 02 18:15:00.157175 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.157158    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"]
Nov 02 18:15:00.157292 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.157276    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"
Nov 02 18:15:00.160583 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.160566    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"
Nov 02 18:15:00.169777 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.169756    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:15:00.172976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.172956    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:15:00.178732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.178622    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"]
Nov 02 18:15:00.183996 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.183968    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5nbb\" (UniqueName: \"kubernetes.io/projected/619ce29c-ca91-4c46-845d-ca06d76ed75c-kube-api-access-f5nbb\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m\" (UID: \"619ce29c-ca91-4c46-845d-ca06d76ed75c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"
Nov 02 18:15:00.184210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.184188    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619ce29c-ca91-4c46-845d-ca06d76ed75c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m\" (UID: \"619ce29c-ca91-4c46-845d-ca06d76ed75c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"
Nov 02 18:15:00.184370 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.184351    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-nhdfk\" (UniqueName: \"kubernetes.io/projected/ecebb449-4550-4812-b86d-96581d77799c-kube-api-access-nhdfk\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk\" (UID: \"ecebb449-4550-4812-b86d-96581d77799c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"
Nov 02 18:15:00.184590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.184572    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecebb449-4550-4812-b86d-96581d77799c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk\" (UID: \"ecebb449-4550-4812-b86d-96581d77799c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"
Nov 02 18:15:00.184767 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.184746    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74ae509-2fe7-4a45-a497-4e7383edbe9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894055-wzr2m\" (UID: \"f74ae509-2fe7-4a45-a497-4e7383edbe9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"
Nov 02 18:15:00.184918 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.184901    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bvhhl\" (UniqueName: \"kubernetes.io/projected/f74ae509-2fe7-4a45-a497-4e7383edbe9a-kube-api-access-bvhhl\") pod \"maas-api-key-cleanup-29894055-wzr2m\" (UID: \"f74ae509-2fe7-4a45-a497-4e7383edbe9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"
Nov 02 18:15:00.185608 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.185584    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:15:00.194129 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.194100    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"]
Nov 02 18:15:00.203472 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.203440    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"]
Nov 02 18:15:00.286056 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.286014    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecebb449-4550-4812-b86d-96581d77799c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk\" (UID: \"ecebb449-4550-4812-b86d-96581d77799c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"
Nov 02 18:15:00.286056 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.286068    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74ae509-2fe7-4a45-a497-4e7383edbe9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894055-wzr2m\" (UID: \"f74ae509-2fe7-4a45-a497-4e7383edbe9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"
Nov 02 18:15:00.286318 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.286092    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bvhhl\" (UniqueName: \"kubernetes.io/projected/f74ae509-2fe7-4a45-a497-4e7383edbe9a-kube-api-access-bvhhl\") pod \"maas-api-key-cleanup-29894055-wzr2m\" (UID: \"f74ae509-2fe7-4a45-a497-4e7383edbe9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"
Nov 02 18:15:00.286318 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.286131    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5nbb\" (UniqueName: \"kubernetes.io/projected/619ce29c-ca91-4c46-845d-ca06d76ed75c-kube-api-access-f5nbb\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m\" (UID: \"619ce29c-ca91-4c46-845d-ca06d76ed75c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"
Nov 02 18:15:00.286441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.286392    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619ce29c-ca91-4c46-845d-ca06d76ed75c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m\" (UID: \"619ce29c-ca91-4c46-845d-ca06d76ed75c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"
Nov 02 18:15:00.286500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.286467    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-nhdfk\" (UniqueName: \"kubernetes.io/projected/ecebb449-4550-4812-b86d-96581d77799c-kube-api-access-nhdfk\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk\" (UID: \"ecebb449-4550-4812-b86d-96581d77799c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"
Nov 02 18:15:00.286854 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.286832    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74ae509-2fe7-4a45-a497-4e7383edbe9a-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894055-wzr2m\" (UID: \"f74ae509-2fe7-4a45-a497-4e7383edbe9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"
Nov 02 18:15:00.286947 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.286927    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecebb449-4550-4812-b86d-96581d77799c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk\" (UID: \"ecebb449-4550-4812-b86d-96581d77799c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"
Nov 02 18:15:00.287038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.287017    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619ce29c-ca91-4c46-845d-ca06d76ed75c-maas-api-service-ca\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m\" (UID: \"619ce29c-ca91-4c46-845d-ca06d76ed75c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"
Nov 02 18:15:00.312525 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.312495    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5nbb\" (UniqueName: \"kubernetes.io/projected/619ce29c-ca91-4c46-845d-ca06d76ed75c-kube-api-access-f5nbb\") pod \"maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m\" (UID: \"619ce29c-ca91-4c46-845d-ca06d76ed75c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"
Nov 02 18:15:00.312634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.312545    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-nhdfk\" (UniqueName: \"kubernetes.io/projected/ecebb449-4550-4812-b86d-96581d77799c-kube-api-access-nhdfk\") pod \"maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk\" (UID: \"ecebb449-4550-4812-b86d-96581d77799c\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"
Nov 02 18:15:00.312672 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.312650    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bvhhl\" (UniqueName: \"kubernetes.io/projected/f74ae509-2fe7-4a45-a497-4e7383edbe9a-kube-api-access-bvhhl\") pod \"maas-api-key-cleanup-29894055-wzr2m\" (UID: \"f74ae509-2fe7-4a45-a497-4e7383edbe9a\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"
Nov 02 18:15:00.468410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.468380    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"
Nov 02 18:15:00.477520 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.477499    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"
Nov 02 18:15:00.509709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.509644    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"
Nov 02 18:15:00.685431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.685396    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:00.776885 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.776857    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"]
Nov 02 18:15:00.777312 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:15:00.777287    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podecebb449_4550_4812_b86d_96581d77799c.slice/crio-e22febb1a4a140a7bed8fddd43593d13ee4e5de020f0def07adf52870196c511 WatchSource:0}: Error finding container e22febb1a4a140a7bed8fddd43593d13ee4e5de020f0def07adf52870196c511: Status 404 returned error can't find the container with id e22febb1a4a140a7bed8fddd43593d13ee4e5de020f0def07adf52870196c511
Nov 02 18:15:00.802204 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.802177    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"]
Nov 02 18:15:00.804228 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:15:00.804197    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podf74ae509_2fe7_4a45_a497_4e7383edbe9a.slice/crio-70bda0266fde7663425e5b8cfca1e112a97654fbc07dfdde399f0f5e0dfbeadc WatchSource:0}: Error finding container 70bda0266fde7663425e5b8cfca1e112a97654fbc07dfdde399f0f5e0dfbeadc: Status 404 returned error can't find the container with id 70bda0266fde7663425e5b8cfca1e112a97654fbc07dfdde399f0f5e0dfbeadc
Nov 02 18:15:00.839513 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:00.839486    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"]
Nov 02 18:15:00.840980 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:15:00.840957    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619ce29c_ca91_4c46_845d_ca06d76ed75c.slice/crio-d800294ffc4fd7ef2391dd5de9d210ac3a8049537090265339aabd0657e0b960 WatchSource:0}: Error finding container d800294ffc4fd7ef2391dd5de9d210ac3a8049537090265339aabd0657e0b960: Status 404 returned error can't find the container with id d800294ffc4fd7ef2391dd5de9d210ac3a8049537090265339aabd0657e0b960
Nov 02 18:15:01.723285 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:01.723258    4456 generic.go:356] "Generic (PLEG): container finished" podID="f74ae509-2fe7-4a45-a497-4e7383edbe9a" containerID="bb054b3ee7d7a9fc31f924c710a24659441ef7ada72abe419984fd50b1d69a44" exitCode=60
Nov 02 18:15:01.723625 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:01.723341    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m" event={"ID":"f74ae509-2fe7-4a45-a497-4e7383edbe9a","Type":"ContainerDied","Data":"bb054b3ee7d7a9fc31f924c710a24659441ef7ada72abe419984fd50b1d69a44"}
Nov 02 18:15:01.723625 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:01.723411    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m" event={"ID":"f74ae509-2fe7-4a45-a497-4e7383edbe9a","Type":"ContainerStarted","Data":"70bda0266fde7663425e5b8cfca1e112a97654fbc07dfdde399f0f5e0dfbeadc"}
Nov 02 18:15:01.723625 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:01.723606    4456 scope.go:122] "RemoveContainer" containerID="bb054b3ee7d7a9fc31f924c710a24659441ef7ada72abe419984fd50b1d69a44"
Nov 02 18:15:01.725493 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:01.725472    4456 generic.go:356] "Generic (PLEG): container finished" podID="619ce29c-ca91-4c46-845d-ca06d76ed75c" containerID="403585014f14c28597c8ed6ed739ec2788ccba6c0e37c89a4abe15916a6a33e2" exitCode=60
Nov 02 18:15:01.725586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:01.725556    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m" event={"ID":"619ce29c-ca91-4c46-845d-ca06d76ed75c","Type":"ContainerDied","Data":"403585014f14c28597c8ed6ed739ec2788ccba6c0e37c89a4abe15916a6a33e2"}
Nov 02 18:15:01.725647 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:01.725615    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m" event={"ID":"619ce29c-ca91-4c46-845d-ca06d76ed75c","Type":"ContainerStarted","Data":"d800294ffc4fd7ef2391dd5de9d210ac3a8049537090265339aabd0657e0b960"}
Nov 02 18:15:01.725762 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:01.725746    4456 scope.go:122] "RemoveContainer" containerID="403585014f14c28597c8ed6ed739ec2788ccba6c0e37c89a4abe15916a6a33e2"
Nov 02 18:15:01.727354 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:01.727332    4456 generic.go:356] "Generic (PLEG): container finished" podID="ecebb449-4550-4812-b86d-96581d77799c" containerID="265e449b83bf5ae428d2988b19feacea5e6bfd6a8e61991422d2b5f9e639d87a" exitCode=60
Nov 02 18:15:01.727441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:01.727353    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk" event={"ID":"ecebb449-4550-4812-b86d-96581d77799c","Type":"ContainerDied","Data":"265e449b83bf5ae428d2988b19feacea5e6bfd6a8e61991422d2b5f9e639d87a"}
Nov 02 18:15:01.727441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:01.727391    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk" event={"ID":"ecebb449-4550-4812-b86d-96581d77799c","Type":"ContainerStarted","Data":"e22febb1a4a140a7bed8fddd43593d13ee4e5de020f0def07adf52870196c511"}
Nov 02 18:15:01.727641 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:01.727622    4456 scope.go:122] "RemoveContainer" containerID="265e449b83bf5ae428d2988b19feacea5e6bfd6a8e61991422d2b5f9e639d87a"
Nov 02 18:15:02.479515 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:02.479487    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:02.621274 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:15:02.621243    4456 cadvisor_stats_provider.go:569] "Partial failure issuing cadvisor.ContainerInfoV2" err="partial failures: [\"/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod619ce29c_ca91_4c46_845d_ca06d76ed75c.slice/crio-a7d0807df2e4b4316c3e7ba39e72894bef0d0554d967f60add65b711f55fafff.scope\": RecentStats: unable to find data in memory cache]"
Nov 02 18:15:02.737332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:02.737244    4456 generic.go:356] "Generic (PLEG): container finished" podID="619ce29c-ca91-4c46-845d-ca06d76ed75c" containerID="a7d0807df2e4b4316c3e7ba39e72894bef0d0554d967f60add65b711f55fafff" exitCode=60
Nov 02 18:15:02.737332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:02.737296    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m" event={"ID":"619ce29c-ca91-4c46-845d-ca06d76ed75c","Type":"ContainerDied","Data":"a7d0807df2e4b4316c3e7ba39e72894bef0d0554d967f60add65b711f55fafff"}
Nov 02 18:15:02.737846 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:02.737358    4456 scope.go:122] "RemoveContainer" containerID="403585014f14c28597c8ed6ed739ec2788ccba6c0e37c89a4abe15916a6a33e2"
Nov 02 18:15:02.737846 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:02.737599    4456 scope.go:122] "RemoveContainer" containerID="a7d0807df2e4b4316c3e7ba39e72894bef0d0554d967f60add65b711f55fafff"
Nov 02 18:15:02.737966 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:15:02.737903    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m_odh-ai-gateway-infra(619ce29c-ca91-4c46-845d-ca06d76ed75c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m" podUID="619ce29c-ca91-4c46-845d-ca06d76ed75c"
Nov 02 18:15:02.739717 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:02.739694    4456 generic.go:356] "Generic (PLEG): container finished" podID="ecebb449-4550-4812-b86d-96581d77799c" containerID="5dc68669f8cc61b881e4bd297614ad0cc82dbd60d9862df7d95bd20466930baa" exitCode=60
Nov 02 18:15:02.739825 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:02.739776    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk" event={"ID":"ecebb449-4550-4812-b86d-96581d77799c","Type":"ContainerDied","Data":"5dc68669f8cc61b881e4bd297614ad0cc82dbd60d9862df7d95bd20466930baa"}
Nov 02 18:15:02.740093 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:02.740072    4456 scope.go:122] "RemoveContainer" containerID="5dc68669f8cc61b881e4bd297614ad0cc82dbd60d9862df7d95bd20466930baa"
Nov 02 18:15:02.740367 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:15:02.740345    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk_odh-ai-gateway-infra(ecebb449-4550-4812-b86d-96581d77799c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk" podUID="ecebb449-4550-4812-b86d-96581d77799c"
Nov 02 18:15:02.742227 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:02.742201    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m" event={"ID":"f74ae509-2fe7-4a45-a497-4e7383edbe9a","Type":"ContainerStarted","Data":"a4b7b907e52d9862d232f9ee58ba56e833f40fb12f38a78f5095be0c8b1745f5"}
Nov 02 18:15:02.753048 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:02.753017    4456 scope.go:122] "RemoveContainer" containerID="265e449b83bf5ae428d2988b19feacea5e6bfd6a8e61991422d2b5f9e639d87a"
Nov 02 18:15:02.865304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:02.865252    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m" podStartSLOduration=2.865238042 podStartE2EDuration="2.865238042s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:00 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:02.863386181 +0000 UTC m=+2719.354449342" watchObservedRunningTime="2026-11-02 18:15:02.865238042 +0000 UTC m=+2719.356301221"
Nov 02 18:15:03.750381 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:03.750339    4456 generic.go:356] "Generic (PLEG): container finished" podID="f74ae509-2fe7-4a45-a497-4e7383edbe9a" containerID="a4b7b907e52d9862d232f9ee58ba56e833f40fb12f38a78f5095be0c8b1745f5" exitCode=60
Nov 02 18:15:03.750856 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:03.750438    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m" event={"ID":"f74ae509-2fe7-4a45-a497-4e7383edbe9a","Type":"ContainerDied","Data":"a4b7b907e52d9862d232f9ee58ba56e833f40fb12f38a78f5095be0c8b1745f5"}
Nov 02 18:15:03.750856 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:03.750493    4456 scope.go:122] "RemoveContainer" containerID="bb054b3ee7d7a9fc31f924c710a24659441ef7ada72abe419984fd50b1d69a44"
Nov 02 18:15:03.750856 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:03.750625    4456 scope.go:122] "RemoveContainer" containerID="a4b7b907e52d9862d232f9ee58ba56e833f40fb12f38a78f5095be0c8b1745f5"
Nov 02 18:15:03.751040 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:15:03.750917    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894055-wzr2m_odh-ai-gateway-infra(f74ae509-2fe7-4a45-a497-4e7383edbe9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m" podUID="f74ae509-2fe7-4a45-a497-4e7383edbe9a"
Nov 02 18:15:03.752605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:03.752578    4456 scope.go:122] "RemoveContainer" containerID="a7d0807df2e4b4316c3e7ba39e72894bef0d0554d967f60add65b711f55fafff"
Nov 02 18:15:03.752849 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:15:03.752805    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m_odh-ai-gateway-infra(619ce29c-ca91-4c46-845d-ca06d76ed75c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m" podUID="619ce29c-ca91-4c46-845d-ca06d76ed75c"
Nov 02 18:15:03.754142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:03.754126    4456 scope.go:122] "RemoveContainer" containerID="5dc68669f8cc61b881e4bd297614ad0cc82dbd60d9862df7d95bd20466930baa"
Nov 02 18:15:03.754319 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:15:03.754302    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk_odh-ai-gateway-infra(ecebb449-4550-4812-b86d-96581d77799c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk" podUID="ecebb449-4550-4812-b86d-96581d77799c"
Nov 02 18:15:04.250045 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:04.250010    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:04.762404 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:04.762377    4456 scope.go:122] "RemoveContainer" containerID="a4b7b907e52d9862d232f9ee58ba56e833f40fb12f38a78f5095be0c8b1745f5"
Nov 02 18:15:04.762786 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:15:04.762620    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894055-wzr2m_odh-ai-gateway-infra(f74ae509-2fe7-4a45-a497-4e7383edbe9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m" podUID="f74ae509-2fe7-4a45-a497-4e7383edbe9a"
Nov 02 18:15:05.302383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:05.302342    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:05.503372 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:05.503340    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:06.053797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:06.053761    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:07.734078 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:07.733993    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:09.355252 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.355215    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24" podUID="3a915d83-872d-4022-9e74-f2fa93aeb737" containerName="main" containerID="cri-o://3878b633a8eaa74e610fcee33456a6cf5b3c7971162acca80149d55649d705c0" gracePeriod=45
Nov 02 18:15:09.486794 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.486766    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:09.633358 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.633290    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:15:09.687564 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.687526    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3a915d83-872d-4022-9e74-f2fa93aeb737-kube-api-access-zzqkn\" (UniqueName: \"kubernetes.io/projected/3a915d83-872d-4022-9e74-f2fa93aeb737-kube-api-access-zzqkn\") pod \"3a915d83-872d-4022-9e74-f2fa93aeb737\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") "
Nov 02 18:15:09.687755 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.687626    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-tmp-dir\") pod \"3a915d83-872d-4022-9e74-f2fa93aeb737\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") "
Nov 02 18:15:09.687755 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.687668    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3a915d83-872d-4022-9e74-f2fa93aeb737-tls-certs\" (UniqueName: \"kubernetes.io/secret/3a915d83-872d-4022-9e74-f2fa93aeb737-tls-certs\") pod \"3a915d83-872d-4022-9e74-f2fa93aeb737\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") "
Nov 02 18:15:09.687911 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.687787    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-model-cache\") pod \"3a915d83-872d-4022-9e74-f2fa93aeb737\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") "
Nov 02 18:15:09.687911 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.687894    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-dshm\") pod \"3a915d83-872d-4022-9e74-f2fa93aeb737\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") "
Nov 02 18:15:09.688024 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.687940    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-home\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-home\") pod \"3a915d83-872d-4022-9e74-f2fa93aeb737\" (UID: \"3a915d83-872d-4022-9e74-f2fa93aeb737\") "
Nov 02 18:15:09.688394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.688363    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-model-cache" pod "3a915d83-872d-4022-9e74-f2fa93aeb737" (UID: "3a915d83-872d-4022-9e74-f2fa93aeb737"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:09.688723 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.688683    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-home" pod "3a915d83-872d-4022-9e74-f2fa93aeb737" (UID: "3a915d83-872d-4022-9e74-f2fa93aeb737"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:09.688950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.688925    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-tmp-dir" pod "3a915d83-872d-4022-9e74-f2fa93aeb737" (UID: "3a915d83-872d-4022-9e74-f2fa93aeb737"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:09.691455 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.691391    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3a915d83-872d-4022-9e74-f2fa93aeb737-kube-api-access-zzqkn" pod "3a915d83-872d-4022-9e74-f2fa93aeb737" (UID: "3a915d83-872d-4022-9e74-f2fa93aeb737"). InnerVolumeSpecName "kube-api-access-zzqkn". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:09.691455 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.691413    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3a915d83-872d-4022-9e74-f2fa93aeb737-tls-certs" pod "3a915d83-872d-4022-9e74-f2fa93aeb737" (UID: "3a915d83-872d-4022-9e74-f2fa93aeb737"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:15:09.711961 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.711937    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-dshm" pod "3a915d83-872d-4022-9e74-f2fa93aeb737" (UID: "3a915d83-872d-4022-9e74-f2fa93aeb737"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:09.789481 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.789450    4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-model-cache\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:09.789605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.789486    4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-dshm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:09.789605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.789498    4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-home\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:09.789605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.789512    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zzqkn\" (UniqueName: \"kubernetes.io/projected/3a915d83-872d-4022-9e74-f2fa93aeb737-kube-api-access-zzqkn\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:09.789605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.789528    4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/3a915d83-872d-4022-9e74-f2fa93aeb737-tmp-dir\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:09.789605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.789540    4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/3a915d83-872d-4022-9e74-f2fa93aeb737-tls-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:09.798577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.798546    4456 generic.go:356] "Generic (PLEG): container finished" podID="3a915d83-872d-4022-9e74-f2fa93aeb737" containerID="3878b633a8eaa74e610fcee33456a6cf5b3c7971162acca80149d55649d705c0" exitCode=0
Nov 02 18:15:09.798685 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.798607    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"
Nov 02 18:15:09.798685 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.798645    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24" event={"ID":"3a915d83-872d-4022-9e74-f2fa93aeb737","Type":"ContainerDied","Data":"3878b633a8eaa74e610fcee33456a6cf5b3c7971162acca80149d55649d705c0"}
Nov 02 18:15:09.798771 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.798684    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24" event={"ID":"3a915d83-872d-4022-9e74-f2fa93aeb737","Type":"ContainerDied","Data":"ac874e35b60759f8b57acad698f1b105105d0891f501fe769320f05ff826eea4"}
Nov 02 18:15:09.798771 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.798702    4456 scope.go:122] "RemoveContainer" containerID="3878b633a8eaa74e610fcee33456a6cf5b3c7971162acca80149d55649d705c0"
Nov 02 18:15:09.809211 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.809190    4456 scope.go:122] "RemoveContainer" containerID="3878b633a8eaa74e610fcee33456a6cf5b3c7971162acca80149d55649d705c0"
Nov 02 18:15:09.809548 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:15:09.809518    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3878b633a8eaa74e610fcee33456a6cf5b3c7971162acca80149d55649d705c0\": container with ID starting with 3878b633a8eaa74e610fcee33456a6cf5b3c7971162acca80149d55649d705c0 not found: ID does not exist" containerID="3878b633a8eaa74e610fcee33456a6cf5b3c7971162acca80149d55649d705c0"
Nov 02 18:15:09.809645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.809602    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3878b633a8eaa74e610fcee33456a6cf5b3c7971162acca80149d55649d705c0"} err="failed to get container status \"3878b633a8eaa74e610fcee33456a6cf5b3c7971162acca80149d55649d705c0\": rpc error: code = NotFound desc = could not find container \"3878b633a8eaa74e610fcee33456a6cf5b3c7971162acca80149d55649d705c0\": container with ID starting with 3878b633a8eaa74e610fcee33456a6cf5b3c7971162acca80149d55649d705c0 not found: ID does not exist"
Nov 02 18:15:09.854449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.854426    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"]
Nov 02 18:15:09.860615 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:09.860592    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24"]
Nov 02 18:15:10.563107 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:10.563067    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="ai-tenant-e2e-shared-b-w0-c4dbe3/auth-test-model-c4dbe3-kserve-65c444678f-vwv24" podUID="3a915d83-872d-4022-9e74-f2fa93aeb737" containerName="main" probeResult="failure" output="Get \"https://10.134.0.255:8000/ready\": net/http: request canceled while waiting for connection (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:15:11.025799 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:11.025766    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:11.612794 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:11.612763    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3a915d83-872d-4022-9e74-f2fa93aeb737" path="/var/lib/kubelet/pods/3a915d83-872d-4022-9e74-f2fa93aeb737/volumes"
Nov 02 18:15:11.619380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:11.619349    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:11.718275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:11.718245    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"]
Nov 02 18:15:11.730770 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:11.730739    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-a-w0-f91b2d-27zxn"]
Nov 02 18:15:11.768565 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:11.768525    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:12.281327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:12.281296    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:13.614654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:13.614621    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9f1070de-7aa8-453e-b11e-17fc5fa3a1d5" path="/var/lib/kubelet/pods/9f1070de-7aa8-453e-b11e-17fc5fa3a1d5/volumes"
Nov 02 18:15:13.649344 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:13.649272    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:14.607114 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:14.607088    4456 scope.go:122] "RemoveContainer" containerID="a4b7b907e52d9862d232f9ee58ba56e833f40fb12f38a78f5095be0c8b1745f5"
Nov 02 18:15:15.352734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:15.352701    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:15.607300 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:15.607227    4456 scope.go:122] "RemoveContainer" containerID="5dc68669f8cc61b881e4bd297614ad0cc82dbd60d9862df7d95bd20466930baa"
Nov 02 18:15:15.843743 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:15.843709    4456 generic.go:356] "Generic (PLEG): container finished" podID="f74ae509-2fe7-4a45-a497-4e7383edbe9a" containerID="42a2f60feeea6f015dc9ef1d04f098d1662025ebedd655fd7f63447cbedec3c2" exitCode=60
Nov 02 18:15:15.843924 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:15.843747    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m" event={"ID":"f74ae509-2fe7-4a45-a497-4e7383edbe9a","Type":"ContainerDied","Data":"42a2f60feeea6f015dc9ef1d04f098d1662025ebedd655fd7f63447cbedec3c2"}
Nov 02 18:15:15.843924 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:15.843806    4456 scope.go:122] "RemoveContainer" containerID="a4b7b907e52d9862d232f9ee58ba56e833f40fb12f38a78f5095be0c8b1745f5"
Nov 02 18:15:15.844242 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:15.844214    4456 scope.go:122] "RemoveContainer" containerID="42a2f60feeea6f015dc9ef1d04f098d1662025ebedd655fd7f63447cbedec3c2"
Nov 02 18:15:15.844543 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:15:15.844510    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894055-wzr2m_odh-ai-gateway-infra(f74ae509-2fe7-4a45-a497-4e7383edbe9a)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m" podUID="f74ae509-2fe7-4a45-a497-4e7383edbe9a"
Nov 02 18:15:16.165720 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:15:16.165640    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:15:16.606867 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:16.606828    4456 scope.go:122] "RemoveContainer" containerID="a7d0807df2e4b4316c3e7ba39e72894bef0d0554d967f60add65b711f55fafff"
Nov 02 18:15:16.854011 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:16.853975    4456 generic.go:356] "Generic (PLEG): container finished" podID="ecebb449-4550-4812-b86d-96581d77799c" containerID="30a84b2e95bc772b5892b79a5369e3e8dc13ebbf200a00ab39a2eb284464efe4" exitCode=60
Nov 02 18:15:16.854181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:16.854110    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk" event={"ID":"ecebb449-4550-4812-b86d-96581d77799c","Type":"ContainerDied","Data":"30a84b2e95bc772b5892b79a5369e3e8dc13ebbf200a00ab39a2eb284464efe4"}
Nov 02 18:15:16.854181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:16.854153    4456 scope.go:122] "RemoveContainer" containerID="5dc68669f8cc61b881e4bd297614ad0cc82dbd60d9862df7d95bd20466930baa"
Nov 02 18:15:16.854579 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:16.854561    4456 scope.go:122] "RemoveContainer" containerID="30a84b2e95bc772b5892b79a5369e3e8dc13ebbf200a00ab39a2eb284464efe4"
Nov 02 18:15:16.854845 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:15:16.854807    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk_odh-ai-gateway-infra(ecebb449-4550-4812-b86d-96581d77799c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk" podUID="ecebb449-4550-4812-b86d-96581d77799c"
Nov 02 18:15:16.896493 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:16.896455    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"]
Nov 02 18:15:17.030613 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.030589    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"
Nov 02 18:15:17.140950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.140869    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:17.161008 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.160975    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/f74ae509-2fe7-4a45-a497-4e7383edbe9a-kube-api-access-bvhhl\" (UniqueName: \"kubernetes.io/projected/f74ae509-2fe7-4a45-a497-4e7383edbe9a-kube-api-access-bvhhl\") pod \"f74ae509-2fe7-4a45-a497-4e7383edbe9a\" (UID: \"f74ae509-2fe7-4a45-a497-4e7383edbe9a\") "
Nov 02 18:15:17.161192 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.161121    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/f74ae509-2fe7-4a45-a497-4e7383edbe9a-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74ae509-2fe7-4a45-a497-4e7383edbe9a-maas-api-service-ca\") pod \"f74ae509-2fe7-4a45-a497-4e7383edbe9a\" (UID: \"f74ae509-2fe7-4a45-a497-4e7383edbe9a\") "
Nov 02 18:15:17.161541 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.161511    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/f74ae509-2fe7-4a45-a497-4e7383edbe9a-maas-api-service-ca" pod "f74ae509-2fe7-4a45-a497-4e7383edbe9a" (UID: "f74ae509-2fe7-4a45-a497-4e7383edbe9a"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:17.163529 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.163503    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/f74ae509-2fe7-4a45-a497-4e7383edbe9a-kube-api-access-bvhhl" pod "f74ae509-2fe7-4a45-a497-4e7383edbe9a" (UID: "f74ae509-2fe7-4a45-a497-4e7383edbe9a"). InnerVolumeSpecName "kube-api-access-bvhhl". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:17.261985 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.261954    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bvhhl\" (UniqueName: \"kubernetes.io/projected/f74ae509-2fe7-4a45-a497-4e7383edbe9a-kube-api-access-bvhhl\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:17.261985 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.261980    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/f74ae509-2fe7-4a45-a497-4e7383edbe9a-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:17.861385 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.861351    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"
Nov 02 18:15:17.861894 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.861381    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m" event={"ID":"f74ae509-2fe7-4a45-a497-4e7383edbe9a","Type":"ContainerDied","Data":"70bda0266fde7663425e5b8cfca1e112a97654fbc07dfdde399f0f5e0dfbeadc"}
Nov 02 18:15:17.861894 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.861447    4456 scope.go:122] "RemoveContainer" containerID="42a2f60feeea6f015dc9ef1d04f098d1662025ebedd655fd7f63447cbedec3c2"
Nov 02 18:15:17.863927 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.863899    4456 generic.go:356] "Generic (PLEG): container finished" podID="619ce29c-ca91-4c46-845d-ca06d76ed75c" containerID="ba6e3b6c7ac01a7063db9ccdd0583bddc9c104acc99c64dfe78682bdc40c45ff" exitCode=60
Nov 02 18:15:17.864063 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.863992    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m" event={"ID":"619ce29c-ca91-4c46-845d-ca06d76ed75c","Type":"ContainerDied","Data":"ba6e3b6c7ac01a7063db9ccdd0583bddc9c104acc99c64dfe78682bdc40c45ff"}
Nov 02 18:15:17.864357 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.864340    4456 scope.go:122] "RemoveContainer" containerID="ba6e3b6c7ac01a7063db9ccdd0583bddc9c104acc99c64dfe78682bdc40c45ff"
Nov 02 18:15:17.864576 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:15:17.864556    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m_odh-ai-gateway-infra(619ce29c-ca91-4c46-845d-ca06d76ed75c)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m" podUID="619ce29c-ca91-4c46-845d-ca06d76ed75c"
Nov 02 18:15:17.872298 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.872278    4456 scope.go:122] "RemoveContainer" containerID="a7d0807df2e4b4316c3e7ba39e72894bef0d0554d967f60add65b711f55fafff"
Nov 02 18:15:17.924500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:17.924473    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"]
Nov 02 18:15:18.017933 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:18.017899    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"]
Nov 02 18:15:18.022994 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:18.022964    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894055-wzr2m"]
Nov 02 18:15:18.066264 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:18.066238    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"
Nov 02 18:15:18.171412 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:18.171330    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ecebb449-4550-4812-b86d-96581d77799c-kube-api-access-nhdfk\" (UniqueName: \"kubernetes.io/projected/ecebb449-4550-4812-b86d-96581d77799c-kube-api-access-nhdfk\") pod \"ecebb449-4550-4812-b86d-96581d77799c\" (UID: \"ecebb449-4550-4812-b86d-96581d77799c\") "
Nov 02 18:15:18.171543 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:18.171491    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ecebb449-4550-4812-b86d-96581d77799c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecebb449-4550-4812-b86d-96581d77799c-maas-api-service-ca\") pod \"ecebb449-4550-4812-b86d-96581d77799c\" (UID: \"ecebb449-4550-4812-b86d-96581d77799c\") "
Nov 02 18:15:18.171910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:18.171888    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ecebb449-4550-4812-b86d-96581d77799c-maas-api-service-ca" pod "ecebb449-4550-4812-b86d-96581d77799c" (UID: "ecebb449-4550-4812-b86d-96581d77799c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:18.173566 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:18.173542    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ecebb449-4550-4812-b86d-96581d77799c-kube-api-access-nhdfk" pod "ecebb449-4550-4812-b86d-96581d77799c" (UID: "ecebb449-4550-4812-b86d-96581d77799c"). InnerVolumeSpecName "kube-api-access-nhdfk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:18.272382 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:18.272342    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-nhdfk\" (UniqueName: \"kubernetes.io/projected/ecebb449-4550-4812-b86d-96581d77799c-kube-api-access-nhdfk\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:18.272382 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:18.272380    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/ecebb449-4550-4812-b86d-96581d77799c-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:18.875156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:18.875123    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"
Nov 02 18:15:18.875647 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:18.875155    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk" event={"ID":"ecebb449-4550-4812-b86d-96581d77799c","Type":"ContainerDied","Data":"e22febb1a4a140a7bed8fddd43593d13ee4e5de020f0def07adf52870196c511"}
Nov 02 18:15:18.875647 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:18.875210    4456 scope.go:122] "RemoveContainer" containerID="30a84b2e95bc772b5892b79a5369e3e8dc13ebbf200a00ab39a2eb284464efe4"
Nov 02 18:15:18.939067 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:18.939033    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"]
Nov 02 18:15:18.949633 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:18.949595    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-worker-w0-069fa5-29894055-xc6zk"]
Nov 02 18:15:18.989240 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:18.989209    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"]
Nov 02 18:15:19.005065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.005012    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:19.139174 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.139107    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"
Nov 02 18:15:19.174896 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.174866    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:19.284559 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.284513    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/619ce29c-ca91-4c46-845d-ca06d76ed75c-kube-api-access-f5nbb\" (UniqueName: \"kubernetes.io/projected/619ce29c-ca91-4c46-845d-ca06d76ed75c-kube-api-access-f5nbb\") pod \"619ce29c-ca91-4c46-845d-ca06d76ed75c\" (UID: \"619ce29c-ca91-4c46-845d-ca06d76ed75c\") "
Nov 02 18:15:19.284770 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.284653    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/619ce29c-ca91-4c46-845d-ca06d76ed75c-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619ce29c-ca91-4c46-845d-ca06d76ed75c-maas-api-service-ca\") pod \"619ce29c-ca91-4c46-845d-ca06d76ed75c\" (UID: \"619ce29c-ca91-4c46-845d-ca06d76ed75c\") "
Nov 02 18:15:19.285427 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.285385    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/619ce29c-ca91-4c46-845d-ca06d76ed75c-maas-api-service-ca" pod "619ce29c-ca91-4c46-845d-ca06d76ed75c" (UID: "619ce29c-ca91-4c46-845d-ca06d76ed75c"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:19.289389 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.289362    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/619ce29c-ca91-4c46-845d-ca06d76ed75c-kube-api-access-f5nbb" pod "619ce29c-ca91-4c46-845d-ca06d76ed75c" (UID: "619ce29c-ca91-4c46-845d-ca06d76ed75c"). InnerVolumeSpecName "kube-api-access-f5nbb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:19.304397 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.304130    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"]
Nov 02 18:15:19.304802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.304768    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p" podUID="56f9278f-4f06-4809-9594-cd1b09070c5c" containerName="istio-proxy" containerID="cri-o://3ac6971e7f9a60a87e7a4ec8c4b7e680f887de253334c6c9abaa27988043685b" gracePeriod=30
Nov 02 18:15:19.377417 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.377375    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:19.385881 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.385808    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/619ce29c-ca91-4c46-845d-ca06d76ed75c-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:19.385881 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.385881    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-f5nbb\" (UniqueName: \"kubernetes.io/projected/619ce29c-ca91-4c46-845d-ca06d76ed75c-kube-api-access-f5nbb\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:19.613408 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.613374    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ecebb449-4550-4812-b86d-96581d77799c" path="/var/lib/kubelet/pods/ecebb449-4550-4812-b86d-96581d77799c/volumes"
Nov 02 18:15:19.613771 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.613757    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="f74ae509-2fe7-4a45-a497-4e7383edbe9a" path="/var/lib/kubelet/pods/f74ae509-2fe7-4a45-a497-4e7383edbe9a/volumes"
Nov 02 18:15:19.865173 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865090    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"]
Nov 02 18:15:19.865618 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865597    4456 cpu_manager.go:414] "containerMap: removing container" podUID="619ce29c-ca91-4c46-845d-ca06d76ed75c" containerName="cleanup"
Nov 02 18:15:19.865713 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865620    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619ce29c-ca91-4c46-845d-ca06d76ed75c" containerName="cleanup"
Nov 02 18:15:19.865713 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865637    4456 cpu_manager.go:414] "containerMap: removing container" podUID="619ce29c-ca91-4c46-845d-ca06d76ed75c" containerName="cleanup"
Nov 02 18:15:19.865713 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865644    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619ce29c-ca91-4c46-845d-ca06d76ed75c" containerName="cleanup"
Nov 02 18:15:19.865713 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865657    4456 cpu_manager.go:414] "containerMap: removing container" podUID="ecebb449-4550-4812-b86d-96581d77799c" containerName="cleanup"
Nov 02 18:15:19.865713 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865663    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecebb449-4550-4812-b86d-96581d77799c" containerName="cleanup"
Nov 02 18:15:19.865713 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865671    4456 cpu_manager.go:414] "containerMap: removing container" podUID="619ce29c-ca91-4c46-845d-ca06d76ed75c" containerName="cleanup"
Nov 02 18:15:19.865713 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865679    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="619ce29c-ca91-4c46-845d-ca06d76ed75c" containerName="cleanup"
Nov 02 18:15:19.865713 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865692    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f74ae509-2fe7-4a45-a497-4e7383edbe9a" containerName="cleanup"
Nov 02 18:15:19.865713 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865699    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74ae509-2fe7-4a45-a497-4e7383edbe9a" containerName="cleanup"
Nov 02 18:15:19.866185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865725    4456 cpu_manager.go:414] "containerMap: removing container" podUID="3a915d83-872d-4022-9e74-f2fa93aeb737" containerName="main"
Nov 02 18:15:19.866185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865732    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3a915d83-872d-4022-9e74-f2fa93aeb737" containerName="main"
Nov 02 18:15:19.866185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865747    4456 cpu_manager.go:414] "containerMap: removing container" podUID="ecebb449-4550-4812-b86d-96581d77799c" containerName="cleanup"
Nov 02 18:15:19.866185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865754    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecebb449-4550-4812-b86d-96581d77799c" containerName="cleanup"
Nov 02 18:15:19.866185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865767    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f74ae509-2fe7-4a45-a497-4e7383edbe9a" containerName="cleanup"
Nov 02 18:15:19.866185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865774    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74ae509-2fe7-4a45-a497-4e7383edbe9a" containerName="cleanup"
Nov 02 18:15:19.866185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865894    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecebb449-4550-4812-b86d-96581d77799c" containerName="cleanup"
Nov 02 18:15:19.866185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865909    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74ae509-2fe7-4a45-a497-4e7383edbe9a" containerName="cleanup"
Nov 02 18:15:19.866185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865925    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="619ce29c-ca91-4c46-845d-ca06d76ed75c" containerName="cleanup"
Nov 02 18:15:19.866185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865934    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="619ce29c-ca91-4c46-845d-ca06d76ed75c" containerName="cleanup"
Nov 02 18:15:19.866185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865943    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="619ce29c-ca91-4c46-845d-ca06d76ed75c" containerName="cleanup"
Nov 02 18:15:19.866185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865955    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74ae509-2fe7-4a45-a497-4e7383edbe9a" containerName="cleanup"
Nov 02 18:15:19.866788 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865964    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecebb449-4550-4812-b86d-96581d77799c" containerName="cleanup"
Nov 02 18:15:19.866788 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.865972    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="3a915d83-872d-4022-9e74-f2fa93aeb737" containerName="main"
Nov 02 18:15:19.870467 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.870446    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"
Nov 02 18:15:19.885571 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.885541    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"
Nov 02 18:15:19.886112 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.886072    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m" event={"ID":"619ce29c-ca91-4c46-845d-ca06d76ed75c","Type":"ContainerDied","Data":"d800294ffc4fd7ef2391dd5de9d210ac3a8049537090265339aabd0657e0b960"}
Nov 02 18:15:19.886295 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.886128    4456 scope.go:122] "RemoveContainer" containerID="ba6e3b6c7ac01a7063db9ccdd0583bddc9c104acc99c64dfe78682bdc40c45ff"
Nov 02 18:15:19.889245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.889222    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"]
Nov 02 18:15:19.894218 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.894195    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:15:19.951540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.951512    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"]
Nov 02 18:15:19.955706 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.955678    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-e2e-shared-b-w0-c4dbe3-29894055-nmx4m"]
Nov 02 18:15:19.991075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.991048    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5\" (UID: \"a1ad17ba-66f0-41fb-853d-ec398bd2cc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"
Nov 02 18:15:19.991197 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.991091    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-xdcn9\" (UniqueName: \"kubernetes.io/projected/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-kube-api-access-xdcn9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5\" (UID: \"a1ad17ba-66f0-41fb-853d-ec398bd2cc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"
Nov 02 18:15:19.991197 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:19.991127    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5\" (UID: \"a1ad17ba-66f0-41fb-853d-ec398bd2cc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"
Nov 02 18:15:20.092673 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:20.092641    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5\" (UID: \"a1ad17ba-66f0-41fb-853d-ec398bd2cc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"
Nov 02 18:15:20.092792 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:20.092681    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-xdcn9\" (UniqueName: \"kubernetes.io/projected/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-kube-api-access-xdcn9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5\" (UID: \"a1ad17ba-66f0-41fb-853d-ec398bd2cc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"
Nov 02 18:15:20.092792 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:20.092710    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5\" (UID: \"a1ad17ba-66f0-41fb-853d-ec398bd2cc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"
Nov 02 18:15:20.093297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:20.093272    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5\" (UID: \"a1ad17ba-66f0-41fb-853d-ec398bd2cc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"
Nov 02 18:15:20.095364 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:20.095334    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5\" (UID: \"a1ad17ba-66f0-41fb-853d-ec398bd2cc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"
Nov 02 18:15:20.117622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:20.117570    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-xdcn9\" (UniqueName: \"kubernetes.io/projected/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-kube-api-access-xdcn9\") pod \"maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5\" (UID: \"a1ad17ba-66f0-41fb-853d-ec398bd2cc00\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"
Nov 02 18:15:20.184947 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:20.184922    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"
Nov 02 18:15:20.489883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:20.489858    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"]
Nov 02 18:15:20.491464 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:15:20.491431    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda1ad17ba_66f0_41fb_853d_ec398bd2cc00.slice/crio-77522f371a90799eccd2163c46026afeab5374ac1041fcb71fa14b98f9dceff1 WatchSource:0}: Error finding container 77522f371a90799eccd2163c46026afeab5374ac1041fcb71fa14b98f9dceff1: Status 404 returned error can't find the container with id 77522f371a90799eccd2163c46026afeab5374ac1041fcb71fa14b98f9dceff1
Nov 02 18:15:20.896174 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:20.896088    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5" event={"ID":"a1ad17ba-66f0-41fb-853d-ec398bd2cc00","Type":"ContainerStarted","Data":"0694f718d8f4922ad5511fdd5cbacbe36d42c307f2281447e2e3eb26eb66fe86"}
Nov 02 18:15:20.896174 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:20.896154    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5" event={"ID":"a1ad17ba-66f0-41fb-853d-ec398bd2cc00","Type":"ContainerStarted","Data":"77522f371a90799eccd2163c46026afeab5374ac1041fcb71fa14b98f9dceff1"}
Nov 02 18:15:20.940880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:20.940830    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5" podStartSLOduration=1.940804787 podStartE2EDuration="1.940804787s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:19 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:20.938469395 +0000 UTC m=+2737.429532576" watchObservedRunningTime="2026-11-02 18:15:20.940804787 +0000 UTC m=+2737.431867948"
Nov 02 18:15:20.947136 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:20.947110    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:21.612784 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:21.612686    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="619ce29c-ca91-4c46-845d-ca06d76ed75c" path="/var/lib/kubelet/pods/619ce29c-ca91-4c46-845d-ca06d76ed75c/volumes"
Nov 02 18:15:21.905157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:21.905068    4456 generic.go:356] "Generic (PLEG): container finished" podID="a1ad17ba-66f0-41fb-853d-ec398bd2cc00" containerID="0694f718d8f4922ad5511fdd5cbacbe36d42c307f2281447e2e3eb26eb66fe86" exitCode=0
Nov 02 18:15:21.905553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:21.905161    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5" event={"ID":"a1ad17ba-66f0-41fb-853d-ec398bd2cc00","Type":"ContainerDied","Data":"0694f718d8f4922ad5511fdd5cbacbe36d42c307f2281447e2e3eb26eb66fe86"}
Nov 02 18:15:22.749603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:22.749568    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:23.056873 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:23.056848    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"
Nov 02 18:15:23.224767 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:23.224729    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-kube-api-access-xdcn9\" (UniqueName: \"kubernetes.io/projected/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-kube-api-access-xdcn9\") pod \"a1ad17ba-66f0-41fb-853d-ec398bd2cc00\" (UID: \"a1ad17ba-66f0-41fb-853d-ec398bd2cc00\") "
Nov 02 18:15:23.224953 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:23.224837    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-maas-api-xks-ca\") pod \"a1ad17ba-66f0-41fb-853d-ec398bd2cc00\" (UID: \"a1ad17ba-66f0-41fb-853d-ec398bd2cc00\") "
Nov 02 18:15:23.224953 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:23.224887    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-maas-api-openshift-ca\") pod \"a1ad17ba-66f0-41fb-853d-ec398bd2cc00\" (UID: \"a1ad17ba-66f0-41fb-853d-ec398bd2cc00\") "
Nov 02 18:15:23.225261 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:23.225241    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-maas-api-openshift-ca" pod "a1ad17ba-66f0-41fb-853d-ec398bd2cc00" (UID: "a1ad17ba-66f0-41fb-853d-ec398bd2cc00"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:23.227160 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:23.227122    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-maas-api-xks-ca" pod "a1ad17ba-66f0-41fb-853d-ec398bd2cc00" (UID: "a1ad17ba-66f0-41fb-853d-ec398bd2cc00"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:15:23.227267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:23.227223    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-kube-api-access-xdcn9" pod "a1ad17ba-66f0-41fb-853d-ec398bd2cc00" (UID: "a1ad17ba-66f0-41fb-853d-ec398bd2cc00"). InnerVolumeSpecName "kube-api-access-xdcn9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:23.325641 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:23.325552    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:23.325641 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:23.325581    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-xdcn9\" (UniqueName: \"kubernetes.io/projected/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-kube-api-access-xdcn9\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:23.325641 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:23.325591    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a1ad17ba-66f0-41fb-853d-ec398bd2cc00-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:23.919702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:23.919613    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"
Nov 02 18:15:23.919702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:23.919630    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5" event={"ID":"a1ad17ba-66f0-41fb-853d-ec398bd2cc00","Type":"ContainerDied","Data":"77522f371a90799eccd2163c46026afeab5374ac1041fcb71fa14b98f9dceff1"}
Nov 02 18:15:23.919702 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:23.919683    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="77522f371a90799eccd2163c46026afeab5374ac1041fcb71fa14b98f9dceff1"
Nov 02 18:15:24.462479 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.462451    4456 patch_prober.go:29] interesting pod/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused" start-of-body=
Nov 02 18:15:24.462906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.462500    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p" podUID="56f9278f-4f06-4809-9594-cd1b09070c5c" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.4:15021/healthz/ready\": dial tcp 10.134.1.4:15021: connect: connection refused"
Nov 02 18:15:24.544151 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.544120    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:24.571650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.571624    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:15:24.738544 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.738512    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-envoy\") pod \"56f9278f-4f06-4809-9594-cd1b09070c5c\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") "
Nov 02 18:15:24.738710 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.738577    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/56f9278f-4f06-4809-9594-cd1b09070c5c-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f9278f-4f06-4809-9594-cd1b09070c5c-istiod-ca-cert\") pod \"56f9278f-4f06-4809-9594-cd1b09070c5c\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") "
Nov 02 18:15:24.738710 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.738614    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-workload-certs\") pod \"56f9278f-4f06-4809-9594-cd1b09070c5c\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") "
Nov 02 18:15:24.738710 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.738686    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-data\") pod \"56f9278f-4f06-4809-9594-cd1b09070c5c\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") "
Nov 02 18:15:24.738956 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.738746    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-podinfo\") pod \"56f9278f-4f06-4809-9594-cd1b09070c5c\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") "
Nov 02 18:15:24.738956 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.738781    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-credential-socket\") pod \"56f9278f-4f06-4809-9594-cd1b09070c5c\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") "
Nov 02 18:15:24.738956 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.738809    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f9278f-4f06-4809-9594-cd1b09070c5c-kube-api-access-29dwk\" (UniqueName: \"kubernetes.io/projected/56f9278f-4f06-4809-9594-cd1b09070c5c-kube-api-access-29dwk\") pod \"56f9278f-4f06-4809-9594-cd1b09070c5c\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") "
Nov 02 18:15:24.738956 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.738888    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-workload-socket\") pod \"56f9278f-4f06-4809-9594-cd1b09070c5c\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") "
Nov 02 18:15:24.739153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.738981    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-workload-certs" pod "56f9278f-4f06-4809-9594-cd1b09070c5c" (UID: "56f9278f-4f06-4809-9594-cd1b09070c5c"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:24.739153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.738995    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-data" pod "56f9278f-4f06-4809-9594-cd1b09070c5c" (UID: "56f9278f-4f06-4809-9594-cd1b09070c5c"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:24.739153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.739011    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-token\" (UniqueName: \"kubernetes.io/projected/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-token\") pod \"56f9278f-4f06-4809-9594-cd1b09070c5c\" (UID: \"56f9278f-4f06-4809-9594-cd1b09070c5c\") "
Nov 02 18:15:24.739372 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.739346    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:24.739457 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.739376    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:24.739629 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.739592    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-workload-socket" pod "56f9278f-4f06-4809-9594-cd1b09070c5c" (UID: "56f9278f-4f06-4809-9594-cd1b09070c5c"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:24.739851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.739807    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/56f9278f-4f06-4809-9594-cd1b09070c5c-istiod-ca-cert" pod "56f9278f-4f06-4809-9594-cd1b09070c5c" (UID: "56f9278f-4f06-4809-9594-cd1b09070c5c"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:24.739962 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.739849    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-credential-socket" pod "56f9278f-4f06-4809-9594-cd1b09070c5c" (UID: "56f9278f-4f06-4809-9594-cd1b09070c5c"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:24.741495 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.741327    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-envoy" pod "56f9278f-4f06-4809-9594-cd1b09070c5c" (UID: "56f9278f-4f06-4809-9594-cd1b09070c5c"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:15:24.742035 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.742012    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-podinfo" pod "56f9278f-4f06-4809-9594-cd1b09070c5c" (UID: "56f9278f-4f06-4809-9594-cd1b09070c5c"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:15:24.742101 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.742041    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f9278f-4f06-4809-9594-cd1b09070c5c-kube-api-access-29dwk" pod "56f9278f-4f06-4809-9594-cd1b09070c5c" (UID: "56f9278f-4f06-4809-9594-cd1b09070c5c"). InnerVolumeSpecName "kube-api-access-29dwk". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:24.742101 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.742053    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-token" pod "56f9278f-4f06-4809-9594-cd1b09070c5c" (UID: "56f9278f-4f06-4809-9594-cd1b09070c5c"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:24.840160 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.840127    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:24.840160 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.840154    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:24.840160 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.840163    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/56f9278f-4f06-4809-9594-cd1b09070c5c-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:24.840160 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.840171    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/56f9278f-4f06-4809-9594-cd1b09070c5c-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:24.840414 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.840179    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:24.840414 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.840188    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-29dwk\" (UniqueName: \"kubernetes.io/projected/56f9278f-4f06-4809-9594-cd1b09070c5c-kube-api-access-29dwk\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:24.840414 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.840197    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/56f9278f-4f06-4809-9594-cd1b09070c5c-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:24.927087 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.927053    4456 generic.go:356] "Generic (PLEG): container finished" podID="56f9278f-4f06-4809-9594-cd1b09070c5c" containerID="3ac6971e7f9a60a87e7a4ec8c4b7e680f887de253334c6c9abaa27988043685b" exitCode=0
Nov 02 18:15:24.927232 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.927100    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"
Nov 02 18:15:24.927232 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.927139    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p" event={"ID":"56f9278f-4f06-4809-9594-cd1b09070c5c","Type":"ContainerDied","Data":"3ac6971e7f9a60a87e7a4ec8c4b7e680f887de253334c6c9abaa27988043685b"}
Nov 02 18:15:24.927232 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.927191    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p" event={"ID":"56f9278f-4f06-4809-9594-cd1b09070c5c","Type":"ContainerDied","Data":"25ae22d3a71f0b89370d8fe11d16125c659d9edd89c27c1f618dd98670977f95"}
Nov 02 18:15:24.927232 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.927213    4456 scope.go:122] "RemoveContainer" containerID="3ac6971e7f9a60a87e7a4ec8c4b7e680f887de253334c6c9abaa27988043685b"
Nov 02 18:15:24.937853 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.937830    4456 scope.go:122] "RemoveContainer" containerID="3ac6971e7f9a60a87e7a4ec8c4b7e680f887de253334c6c9abaa27988043685b"
Nov 02 18:15:24.938141 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:15:24.938123    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3ac6971e7f9a60a87e7a4ec8c4b7e680f887de253334c6c9abaa27988043685b\": container with ID starting with 3ac6971e7f9a60a87e7a4ec8c4b7e680f887de253334c6c9abaa27988043685b not found: ID does not exist" containerID="3ac6971e7f9a60a87e7a4ec8c4b7e680f887de253334c6c9abaa27988043685b"
Nov 02 18:15:24.938194 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.938150    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3ac6971e7f9a60a87e7a4ec8c4b7e680f887de253334c6c9abaa27988043685b"} err="failed to get container status \"3ac6971e7f9a60a87e7a4ec8c4b7e680f887de253334c6c9abaa27988043685b\": rpc error: code = NotFound desc = could not find container \"3ac6971e7f9a60a87e7a4ec8c4b7e680f887de253334c6c9abaa27988043685b\": container with ID starting with 3ac6971e7f9a60a87e7a4ec8c4b7e680f887de253334c6c9abaa27988043685b not found: ID does not exist"
Nov 02 18:15:24.983294 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.983259    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"]
Nov 02 18:15:24.996641 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:24.996571    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-shared-a-w0-f91b2d-openshift-default-686b87679-gn58p"]
Nov 02 18:15:25.613033 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:25.613002    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="56f9278f-4f06-4809-9594-cd1b09070c5c" path="/var/lib/kubelet/pods/56f9278f-4f06-4809-9594-cd1b09070c5c/volumes"
Nov 02 18:15:26.250170 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:26.250141    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:27.940253 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:27.940214    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:29.750762 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:29.750726    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:30.305692 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:30.305656    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"]
Nov 02 18:15:30.477157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:30.477124    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:30.616297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:30.616209    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:31.538992 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:31.538958    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:33.057115 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:33.057075    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:33.628204 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:33.628172    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:33.721661 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:33.721631    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"]
Nov 02 18:15:33.734957 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:33.734926    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-shared-b-w0-c4dbe3-ztqn5"]
Nov 02 18:15:33.767950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:33.767918    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:34.229232 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:34.229200    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:35.306899 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:35.306857    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing" containerID="cri-o://1a5f4c3158352df507219d936b2b2df3545243e213952a4a8b0fa5bfff6d9c28" gracePeriod=25
Nov 02 18:15:35.612917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:35.612835    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ad17ba-66f0-41fb-853d-ec398bd2cc00" path="/var/lib/kubelet/pods/a1ad17ba-66f0-41fb-853d-ec398bd2cc00/volumes"
Nov 02 18:15:35.649408 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:35.649372    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:36.980263 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:36.980228    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:37.117947 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:37.117915    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:37.356524 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:37.356438    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:38.391866 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:38.391830    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:39.050554 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:39.050518    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:40.855530 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:40.855491    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:40.997255 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:40.997222    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:41.271406 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.271377    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:41.399534 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.399497    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:15:41.688977 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.688947    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:41.902446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.902413    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"]
Nov 02 18:15:41.902876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.902798    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a1ad17ba-66f0-41fb-853d-ec398bd2cc00" containerName="revoke-keys"
Nov 02 18:15:41.902876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.902809    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ad17ba-66f0-41fb-853d-ec398bd2cc00" containerName="revoke-keys"
Nov 02 18:15:41.902876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.902835    4456 cpu_manager.go:414] "containerMap: removing container" podUID="56f9278f-4f06-4809-9594-cd1b09070c5c" containerName="istio-proxy"
Nov 02 18:15:41.902876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.902840    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="56f9278f-4f06-4809-9594-cd1b09070c5c" containerName="istio-proxy"
Nov 02 18:15:41.902876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.902848    4456 cpu_manager.go:414] "containerMap: removing container" podUID="ecebb449-4550-4812-b86d-96581d77799c" containerName="cleanup"
Nov 02 18:15:41.902876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.902852    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ecebb449-4550-4812-b86d-96581d77799c" containerName="cleanup"
Nov 02 18:15:41.902876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.902861    4456 cpu_manager.go:414] "containerMap: removing container" podUID="f74ae509-2fe7-4a45-a497-4e7383edbe9a" containerName="cleanup"
Nov 02 18:15:41.902876 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.902866    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="f74ae509-2fe7-4a45-a497-4e7383edbe9a" containerName="cleanup"
Nov 02 18:15:41.903119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.902956    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ad17ba-66f0-41fb-853d-ec398bd2cc00" containerName="revoke-keys"
Nov 02 18:15:41.903119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.902964    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="56f9278f-4f06-4809-9594-cd1b09070c5c" containerName="istio-proxy"
Nov 02 18:15:41.903119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.902975    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ecebb449-4550-4812-b86d-96581d77799c" containerName="cleanup"
Nov 02 18:15:41.903119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.902983    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="f74ae509-2fe7-4a45-a497-4e7383edbe9a" containerName="cleanup"
Nov 02 18:15:41.907364 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.907343    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"
Nov 02 18:15:41.923571 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.923551    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:15:41.923695 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.923673    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:15:41.925275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.925253    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"]
Nov 02 18:15:41.935256 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.935221    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:15:41.936940 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:41.936921    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:15:42.017352 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:42.017262    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b3dd050-cbf9-4796-8440-1422c946dfa5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf\" (UID: \"9b3dd050-cbf9-4796-8440-1422c946dfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"
Nov 02 18:15:42.017352 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:42.017321    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bb2sb\" (UniqueName: \"kubernetes.io/projected/9b3dd050-cbf9-4796-8440-1422c946dfa5-kube-api-access-bb2sb\") pod \"maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf\" (UID: \"9b3dd050-cbf9-4796-8440-1422c946dfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"
Nov 02 18:15:42.017522 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:42.017454    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b3dd050-cbf9-4796-8440-1422c946dfa5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf\" (UID: \"9b3dd050-cbf9-4796-8440-1422c946dfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"
Nov 02 18:15:42.119029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:42.118990    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bb2sb\" (UniqueName: \"kubernetes.io/projected/9b3dd050-cbf9-4796-8440-1422c946dfa5-kube-api-access-bb2sb\") pod \"maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf\" (UID: \"9b3dd050-cbf9-4796-8440-1422c946dfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"
Nov 02 18:15:42.119199 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:42.119044    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b3dd050-cbf9-4796-8440-1422c946dfa5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf\" (UID: \"9b3dd050-cbf9-4796-8440-1422c946dfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"
Nov 02 18:15:42.119199 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:42.119125    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b3dd050-cbf9-4796-8440-1422c946dfa5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf\" (UID: \"9b3dd050-cbf9-4796-8440-1422c946dfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"
Nov 02 18:15:42.120275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:42.120241    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b3dd050-cbf9-4796-8440-1422c946dfa5-maas-api-openshift-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf\" (UID: \"9b3dd050-cbf9-4796-8440-1422c946dfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"
Nov 02 18:15:42.129923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:42.129899    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b3dd050-cbf9-4796-8440-1422c946dfa5-maas-api-xks-ca\") pod \"maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf\" (UID: \"9b3dd050-cbf9-4796-8440-1422c946dfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"
Nov 02 18:15:42.144081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:42.144037    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bb2sb\" (UniqueName: \"kubernetes.io/projected/9b3dd050-cbf9-4796-8440-1422c946dfa5-kube-api-access-bb2sb\") pod \"maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf\" (UID: \"9b3dd050-cbf9-4796-8440-1422c946dfa5\") " pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"
Nov 02 18:15:42.220654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:42.220624    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"
Nov 02 18:15:42.523097 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:15:42.523066    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9b3dd050_cbf9_4796_8440_1422c946dfa5.slice/crio-31977bb13fb9b4deeae24fff255f39b6831d16f61827e15926c72732cb69d5b5 WatchSource:0}: Error finding container 31977bb13fb9b4deeae24fff255f39b6831d16f61827e15926c72732cb69d5b5: Status 404 returned error can't find the container with id 31977bb13fb9b4deeae24fff255f39b6831d16f61827e15926c72732cb69d5b5
Nov 02 18:15:42.523494 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:42.523463    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"]
Nov 02 18:15:42.790556 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:42.790523    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:43.049077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:43.048986    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf" event={"ID":"9b3dd050-cbf9-4796-8440-1422c946dfa5","Type":"ContainerStarted","Data":"6676d84ebd251ade95948abf7c8d3e2f18afe551d6430fb4e11b2f461447d2f1"}
Nov 02 18:15:43.049077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:43.049051    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf" event={"ID":"9b3dd050-cbf9-4796-8440-1422c946dfa5","Type":"ContainerStarted","Data":"31977bb13fb9b4deeae24fff255f39b6831d16f61827e15926c72732cb69d5b5"}
Nov 02 18:15:43.107107 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:43.107034    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf" podStartSLOduration=2.107018145 podStartE2EDuration="2.107018145s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:43.091872484 +0000 UTC m=+2759.582935645" watchObservedRunningTime="2026-11-02 18:15:43.107018145 +0000 UTC m=+2759.598081306"
Nov 02 18:15:44.057383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:44.057346    4456 generic.go:356] "Generic (PLEG): container finished" podID="9b3dd050-cbf9-4796-8440-1422c946dfa5" containerID="6676d84ebd251ade95948abf7c8d3e2f18afe551d6430fb4e11b2f461447d2f1" exitCode=0
Nov 02 18:15:44.057833 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:44.057439    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf" event={"ID":"9b3dd050-cbf9-4796-8440-1422c946dfa5","Type":"ContainerDied","Data":"6676d84ebd251ade95948abf7c8d3e2f18afe551d6430fb4e11b2f461447d2f1"}
Nov 02 18:15:44.550829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:44.550777    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:45.203105 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:45.203082    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"
Nov 02 18:15:45.353797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:45.353709    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9b3dd050-cbf9-4796-8440-1422c946dfa5-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b3dd050-cbf9-4796-8440-1422c946dfa5-maas-api-openshift-ca\") pod \"9b3dd050-cbf9-4796-8440-1422c946dfa5\" (UID: \"9b3dd050-cbf9-4796-8440-1422c946dfa5\") "
Nov 02 18:15:45.354000 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:45.353859    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9b3dd050-cbf9-4796-8440-1422c946dfa5-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b3dd050-cbf9-4796-8440-1422c946dfa5-maas-api-xks-ca\") pod \"9b3dd050-cbf9-4796-8440-1422c946dfa5\" (UID: \"9b3dd050-cbf9-4796-8440-1422c946dfa5\") "
Nov 02 18:15:45.354000 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:45.353891    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9b3dd050-cbf9-4796-8440-1422c946dfa5-kube-api-access-bb2sb\" (UniqueName: \"kubernetes.io/projected/9b3dd050-cbf9-4796-8440-1422c946dfa5-kube-api-access-bb2sb\") pod \"9b3dd050-cbf9-4796-8440-1422c946dfa5\" (UID: \"9b3dd050-cbf9-4796-8440-1422c946dfa5\") "
Nov 02 18:15:45.354109 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:45.354068    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9b3dd050-cbf9-4796-8440-1422c946dfa5-maas-api-openshift-ca" pod "9b3dd050-cbf9-4796-8440-1422c946dfa5" (UID: "9b3dd050-cbf9-4796-8440-1422c946dfa5"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:45.354293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:45.354269    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9b3dd050-cbf9-4796-8440-1422c946dfa5-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:45.356209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:45.356176    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9b3dd050-cbf9-4796-8440-1422c946dfa5-kube-api-access-bb2sb" pod "9b3dd050-cbf9-4796-8440-1422c946dfa5" (UID: "9b3dd050-cbf9-4796-8440-1422c946dfa5"). InnerVolumeSpecName "kube-api-access-bb2sb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:45.356209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:45.356201    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9b3dd050-cbf9-4796-8440-1422c946dfa5-maas-api-xks-ca" pod "9b3dd050-cbf9-4796-8440-1422c946dfa5" (UID: "9b3dd050-cbf9-4796-8440-1422c946dfa5"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:15:45.455470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:45.455437    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9b3dd050-cbf9-4796-8440-1422c946dfa5-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:45.455470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:45.455466    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-bb2sb\" (UniqueName: \"kubernetes.io/projected/9b3dd050-cbf9-4796-8440-1422c946dfa5-kube-api-access-bb2sb\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:46.071172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:46.071143    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"
Nov 02 18:15:46.071347 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:46.071175    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf" event={"ID":"9b3dd050-cbf9-4796-8440-1422c946dfa5","Type":"ContainerDied","Data":"31977bb13fb9b4deeae24fff255f39b6831d16f61827e15926c72732cb69d5b5"}
Nov 02 18:15:46.071347 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:46.071221    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="31977bb13fb9b4deeae24fff255f39b6831d16f61827e15926c72732cb69d5b5"
Nov 02 18:15:46.169739 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:15:46.169703    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:15:46.317055 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:46.317023    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:46.627080 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:46.627038    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"]
Nov 02 18:15:46.770588 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:46.770559    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:48.063315 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:48.063266    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:49.762449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:49.762414    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:51.400113 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:51.400079    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:15:51.549924 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:51.549888    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:52.591831 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:52.591768    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:52.732039 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:52.732010    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:53.338226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:53.338193    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:15:54.907908 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:54.907868    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"]
Nov 02 18:15:54.908305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:54.908267    4456 cpu_manager.go:414] "containerMap: removing container" podUID="9b3dd050-cbf9-4796-8440-1422c946dfa5" containerName="revoke-keys"
Nov 02 18:15:54.908305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:54.908279    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9b3dd050-cbf9-4796-8440-1422c946dfa5" containerName="revoke-keys"
Nov 02 18:15:54.908385 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:54.908374    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="9b3dd050-cbf9-4796-8440-1422c946dfa5" containerName="revoke-keys"
Nov 02 18:15:54.911628 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:54.911613    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"
Nov 02 18:15:54.935490 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:54.935458    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"]
Nov 02 18:15:54.943917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:54.943891    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:15:54.944039 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:54.943939    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:15:54.944039 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:54.943950    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:15:54.944039 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:54.943993    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:15:55.046795 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.046764    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/573c6b8d-87c3-486a-8a63-1b6b409103e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4b4dd23-6rhns\" (UID: \"573c6b8d-87c3-486a-8a63-1b6b409103e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"
Nov 02 18:15:55.046963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.046801    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/573c6b8d-87c3-486a-8a63-1b6b409103e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4b4dd23-6rhns\" (UID: \"573c6b8d-87c3-486a-8a63-1b6b409103e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"
Nov 02 18:15:55.046963 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.046934    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-rxgcm\" (UniqueName: \"kubernetes.io/projected/573c6b8d-87c3-486a-8a63-1b6b409103e7-kube-api-access-rxgcm\") pod \"maas-api-revoke-sub-f4b4dd23-6rhns\" (UID: \"573c6b8d-87c3-486a-8a63-1b6b409103e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"
Nov 02 18:15:55.148274 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.148241    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/573c6b8d-87c3-486a-8a63-1b6b409103e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4b4dd23-6rhns\" (UID: \"573c6b8d-87c3-486a-8a63-1b6b409103e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"
Nov 02 18:15:55.148441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.148316    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-rxgcm\" (UniqueName: \"kubernetes.io/projected/573c6b8d-87c3-486a-8a63-1b6b409103e7-kube-api-access-rxgcm\") pod \"maas-api-revoke-sub-f4b4dd23-6rhns\" (UID: \"573c6b8d-87c3-486a-8a63-1b6b409103e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"
Nov 02 18:15:55.148441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.148374    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/573c6b8d-87c3-486a-8a63-1b6b409103e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4b4dd23-6rhns\" (UID: \"573c6b8d-87c3-486a-8a63-1b6b409103e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"
Nov 02 18:15:55.148891 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.148868    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/573c6b8d-87c3-486a-8a63-1b6b409103e7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-f4b4dd23-6rhns\" (UID: \"573c6b8d-87c3-486a-8a63-1b6b409103e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"
Nov 02 18:15:55.150908 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.150880    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/573c6b8d-87c3-486a-8a63-1b6b409103e7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-f4b4dd23-6rhns\" (UID: \"573c6b8d-87c3-486a-8a63-1b6b409103e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"
Nov 02 18:15:55.173416 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.173361    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-rxgcm\" (UniqueName: \"kubernetes.io/projected/573c6b8d-87c3-486a-8a63-1b6b409103e7-kube-api-access-rxgcm\") pod \"maas-api-revoke-sub-f4b4dd23-6rhns\" (UID: \"573c6b8d-87c3-486a-8a63-1b6b409103e7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"
Nov 02 18:15:55.225237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.225215    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"
Nov 02 18:15:55.527620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.527589    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"]
Nov 02 18:15:55.528711 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:15:55.528688    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod573c6b8d_87c3_486a_8a63_1b6b409103e7.slice/crio-1f8162ce3f13c3c195b84a82d6643958b34d3424fcce652e32ca70d6a1e923a2 WatchSource:0}: Error finding container 1f8162ce3f13c3c195b84a82d6643958b34d3424fcce652e32ca70d6a1e923a2: Status 404 returned error can't find the container with id 1f8162ce3f13c3c195b84a82d6643958b34d3424fcce652e32ca70d6a1e923a2
Nov 02 18:15:55.632458 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.632432    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"
Nov 02 18:15:55.753975 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.753943    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/b8847684-5676-4253-850f-9f2b2c42ea0c-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8847684-5676-4253-850f-9f2b2c42ea0c-plugins-config-volume\") pod \"b8847684-5676-4253-850f-9f2b2c42ea0c\" (UID: \"b8847684-5676-4253-850f-9f2b2c42ea0c\") "
Nov 02 18:15:55.754161 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.753983    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/b8847684-5676-4253-850f-9f2b2c42ea0c-kube-api-access-psdvm\" (UniqueName: \"kubernetes.io/projected/b8847684-5676-4253-850f-9f2b2c42ea0c-kube-api-access-psdvm\") pod \"b8847684-5676-4253-850f-9f2b2c42ea0c\" (UID: \"b8847684-5676-4253-850f-9f2b2c42ea0c\") "
Nov 02 18:15:55.754319 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.754294    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/b8847684-5676-4253-850f-9f2b2c42ea0c-plugins-config-volume" pod "b8847684-5676-4253-850f-9f2b2c42ea0c" (UID: "b8847684-5676-4253-850f-9f2b2c42ea0c"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:55.756338 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.756317    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/b8847684-5676-4253-850f-9f2b2c42ea0c-kube-api-access-psdvm" pod "b8847684-5676-4253-850f-9f2b2c42ea0c" (UID: "b8847684-5676-4253-850f-9f2b2c42ea0c"). InnerVolumeSpecName "kube-api-access-psdvm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:55.855452 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.855421    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/b8847684-5676-4253-850f-9f2b2c42ea0c-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:55.855452 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:55.855447    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-psdvm\" (UniqueName: \"kubernetes.io/projected/b8847684-5676-4253-850f-9f2b2c42ea0c-kube-api-access-psdvm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:56.140474 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:56.140392    4456 generic.go:356] "Generic (PLEG): container finished" podID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerID="1a5f4c3158352df507219d936b2b2df3545243e213952a4a8b0fa5bfff6d9c28" exitCode=0
Nov 02 18:15:56.140474 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:56.140437    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"
Nov 02 18:15:56.141087 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:56.140473    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" event={"ID":"b8847684-5676-4253-850f-9f2b2c42ea0c","Type":"ContainerDied","Data":"1a5f4c3158352df507219d936b2b2df3545243e213952a4a8b0fa5bfff6d9c28"}
Nov 02 18:15:56.141087 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:56.140520    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg" event={"ID":"b8847684-5676-4253-850f-9f2b2c42ea0c","Type":"ContainerDied","Data":"0931e1faf1d8a7f8328258d16e252e3b301ac6a4b125a3570dc2b7a0f74640d5"}
Nov 02 18:15:56.141087 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:56.140545    4456 scope.go:122] "RemoveContainer" containerID="1a5f4c3158352df507219d936b2b2df3545243e213952a4a8b0fa5bfff6d9c28"
Nov 02 18:15:56.142653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:56.142629    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns" event={"ID":"573c6b8d-87c3-486a-8a63-1b6b409103e7","Type":"ContainerStarted","Data":"d78d07235afd10e0a4e730bf23edcf9f68d9d0481b60db6d00f9dbc678bacd64"}
Nov 02 18:15:56.142778 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:56.142675    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns" event={"ID":"573c6b8d-87c3-486a-8a63-1b6b409103e7","Type":"ContainerStarted","Data":"1f8162ce3f13c3c195b84a82d6643958b34d3424fcce652e32ca70d6a1e923a2"}
Nov 02 18:15:56.151314 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:56.151296    4456 scope.go:122] "RemoveContainer" containerID="1a5f4c3158352df507219d936b2b2df3545243e213952a4a8b0fa5bfff6d9c28"
Nov 02 18:15:56.151590 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:15:56.151571    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"1a5f4c3158352df507219d936b2b2df3545243e213952a4a8b0fa5bfff6d9c28\": container with ID starting with 1a5f4c3158352df507219d936b2b2df3545243e213952a4a8b0fa5bfff6d9c28 not found: ID does not exist" containerID="1a5f4c3158352df507219d936b2b2df3545243e213952a4a8b0fa5bfff6d9c28"
Nov 02 18:15:56.151656 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:56.151599    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"1a5f4c3158352df507219d936b2b2df3545243e213952a4a8b0fa5bfff6d9c28"} err="failed to get container status \"1a5f4c3158352df507219d936b2b2df3545243e213952a4a8b0fa5bfff6d9c28\": rpc error: code = NotFound desc = could not find container \"1a5f4c3158352df507219d936b2b2df3545243e213952a4a8b0fa5bfff6d9c28\": container with ID starting with 1a5f4c3158352df507219d936b2b2df3545243e213952a4a8b0fa5bfff6d9c28 not found: ID does not exist"
Nov 02 18:15:56.203004 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:56.202962    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns" podStartSLOduration=2.202950074 podStartE2EDuration="2.202950074s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:15:54 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:15:56.19999442 +0000 UTC m=+2772.691057583" watchObservedRunningTime="2026-11-02 18:15:56.202950074 +0000 UTC m=+2772.694013261"
Nov 02 18:15:56.240217 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:56.240188    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"]
Nov 02 18:15:56.244397 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:56.244377    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-shared-b-w0-c4dbe3-8476479769-pdglg"]
Nov 02 18:15:57.151740 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:57.151697    4456 generic.go:356] "Generic (PLEG): container finished" podID="573c6b8d-87c3-486a-8a63-1b6b409103e7" containerID="d78d07235afd10e0a4e730bf23edcf9f68d9d0481b60db6d00f9dbc678bacd64" exitCode=0
Nov 02 18:15:57.152255 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:57.151759    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns" event={"ID":"573c6b8d-87c3-486a-8a63-1b6b409103e7","Type":"ContainerDied","Data":"d78d07235afd10e0a4e730bf23edcf9f68d9d0481b60db6d00f9dbc678bacd64"}
Nov 02 18:15:57.613369 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:57.613335    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" path="/var/lib/kubelet/pods/b8847684-5676-4253-850f-9f2b2c42ea0c/volumes"
Nov 02 18:15:58.311758 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:58.311729    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"
Nov 02 18:15:58.380149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:58.380109    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/573c6b8d-87c3-486a-8a63-1b6b409103e7-kube-api-access-rxgcm\" (UniqueName: \"kubernetes.io/projected/573c6b8d-87c3-486a-8a63-1b6b409103e7-kube-api-access-rxgcm\") pod \"573c6b8d-87c3-486a-8a63-1b6b409103e7\" (UID: \"573c6b8d-87c3-486a-8a63-1b6b409103e7\") "
Nov 02 18:15:58.380334 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:58.380305    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/573c6b8d-87c3-486a-8a63-1b6b409103e7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/573c6b8d-87c3-486a-8a63-1b6b409103e7-maas-api-openshift-ca\") pod \"573c6b8d-87c3-486a-8a63-1b6b409103e7\" (UID: \"573c6b8d-87c3-486a-8a63-1b6b409103e7\") "
Nov 02 18:15:58.380407 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:58.380345    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/573c6b8d-87c3-486a-8a63-1b6b409103e7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/573c6b8d-87c3-486a-8a63-1b6b409103e7-maas-api-xks-ca\") pod \"573c6b8d-87c3-486a-8a63-1b6b409103e7\" (UID: \"573c6b8d-87c3-486a-8a63-1b6b409103e7\") "
Nov 02 18:15:58.380650 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:58.380623    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/573c6b8d-87c3-486a-8a63-1b6b409103e7-maas-api-openshift-ca" pod "573c6b8d-87c3-486a-8a63-1b6b409103e7" (UID: "573c6b8d-87c3-486a-8a63-1b6b409103e7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:15:58.382601 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:58.382576    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/573c6b8d-87c3-486a-8a63-1b6b409103e7-kube-api-access-rxgcm" pod "573c6b8d-87c3-486a-8a63-1b6b409103e7" (UID: "573c6b8d-87c3-486a-8a63-1b6b409103e7"). InnerVolumeSpecName "kube-api-access-rxgcm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:15:58.382925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:58.382893    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/573c6b8d-87c3-486a-8a63-1b6b409103e7-maas-api-xks-ca" pod "573c6b8d-87c3-486a-8a63-1b6b409103e7" (UID: "573c6b8d-87c3-486a-8a63-1b6b409103e7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:15:58.481680 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:58.481640    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-rxgcm\" (UniqueName: \"kubernetes.io/projected/573c6b8d-87c3-486a-8a63-1b6b409103e7-kube-api-access-rxgcm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:58.481680 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:58.481679    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/573c6b8d-87c3-486a-8a63-1b6b409103e7-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:58.481924 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:58.481694    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/573c6b8d-87c3-486a-8a63-1b6b409103e7-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:15:59.167352 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:59.167313    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"
Nov 02 18:15:59.167352 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:59.167343    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns" event={"ID":"573c6b8d-87c3-486a-8a63-1b6b409103e7","Type":"ContainerDied","Data":"1f8162ce3f13c3c195b84a82d6643958b34d3424fcce652e32ca70d6a1e923a2"}
Nov 02 18:15:59.167564 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:15:59.167397    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1f8162ce3f13c3c195b84a82d6643958b34d3424fcce652e32ca70d6a1e923a2"
Nov 02 18:16:01.647549 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.647510    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" containerID="cri-o://d88d0bddc4d93ed42f5f2ffb943fc376a1de597ed4368f492c206bca841a6ef3" gracePeriod=45
Nov 02 18:16:01.730986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.730948    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main" probeResult="failure" output="Get \"https://10.134.0.52:8000/ready\": dial tcp 10.134.0.52:8000: connect: connection refused"
Nov 02 18:16:01.913040 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.912963    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 18:16:01.924514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.924479    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-model-cache\") pod \"9e85cf3f-eb7a-452b-b716-26df5b391065\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") "
Nov 02 18:16:01.924657 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.924540    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9e85cf3f-eb7a-452b-b716-26df5b391065-kube-api-access-wxhm9\" (UniqueName: \"kubernetes.io/projected/9e85cf3f-eb7a-452b-b716-26df5b391065-kube-api-access-wxhm9\") pod \"9e85cf3f-eb7a-452b-b716-26df5b391065\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") "
Nov 02 18:16:01.924657 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.924571    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-home\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-home\") pod \"9e85cf3f-eb7a-452b-b716-26df5b391065\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") "
Nov 02 18:16:01.924657 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.924631    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-tmp-dir\") pod \"9e85cf3f-eb7a-452b-b716-26df5b391065\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") "
Nov 02 18:16:01.924844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.924667    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-dshm\") pod \"9e85cf3f-eb7a-452b-b716-26df5b391065\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") "
Nov 02 18:16:01.924844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.924712    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9e85cf3f-eb7a-452b-b716-26df5b391065-tls-certs\" (UniqueName: \"kubernetes.io/secret/9e85cf3f-eb7a-452b-b716-26df5b391065-tls-certs\") pod \"9e85cf3f-eb7a-452b-b716-26df5b391065\" (UID: \"9e85cf3f-eb7a-452b-b716-26df5b391065\") "
Nov 02 18:16:01.924953 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.924800    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-model-cache" pod "9e85cf3f-eb7a-452b-b716-26df5b391065" (UID: "9e85cf3f-eb7a-452b-b716-26df5b391065"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:01.924953 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.924858    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-home" pod "9e85cf3f-eb7a-452b-b716-26df5b391065" (UID: "9e85cf3f-eb7a-452b-b716-26df5b391065"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:01.924953 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.924903    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-tmp-dir" pod "9e85cf3f-eb7a-452b-b716-26df5b391065" (UID: "9e85cf3f-eb7a-452b-b716-26df5b391065"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:01.925216 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.925192    4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-tmp-dir\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:16:01.925344 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.925271    4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-model-cache\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:16:01.925406 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.925397    4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-home\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:16:01.927013 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.926988    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9e85cf3f-eb7a-452b-b716-26df5b391065-kube-api-access-wxhm9" pod "9e85cf3f-eb7a-452b-b716-26df5b391065" (UID: "9e85cf3f-eb7a-452b-b716-26df5b391065"). InnerVolumeSpecName "kube-api-access-wxhm9". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:16:01.927519 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.927498    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-dshm" pod "9e85cf3f-eb7a-452b-b716-26df5b391065" (UID: "9e85cf3f-eb7a-452b-b716-26df5b391065"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:16:01.927612 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:01.927558    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9e85cf3f-eb7a-452b-b716-26df5b391065-tls-certs" pod "9e85cf3f-eb7a-452b-b716-26df5b391065" (UID: "9e85cf3f-eb7a-452b-b716-26df5b391065"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:16:02.025925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:02.025895    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-wxhm9\" (UniqueName: \"kubernetes.io/projected/9e85cf3f-eb7a-452b-b716-26df5b391065-kube-api-access-wxhm9\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:16:02.025925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:02.025923    4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/9e85cf3f-eb7a-452b-b716-26df5b391065-dshm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:16:02.025925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:02.025932    4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/9e85cf3f-eb7a-452b-b716-26df5b391065-tls-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:16:02.188845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:02.188783    4456 generic.go:356] "Generic (PLEG): container finished" podID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerID="d88d0bddc4d93ed42f5f2ffb943fc376a1de597ed4368f492c206bca841a6ef3" exitCode=0
Nov 02 18:16:02.188998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:02.188859    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"
Nov 02 18:16:02.188998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:02.188858    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" event={"ID":"9e85cf3f-eb7a-452b-b716-26df5b391065","Type":"ContainerDied","Data":"d88d0bddc4d93ed42f5f2ffb943fc376a1de597ed4368f492c206bca841a6ef3"}
Nov 02 18:16:02.188998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:02.188972    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr" event={"ID":"9e85cf3f-eb7a-452b-b716-26df5b391065","Type":"ContainerDied","Data":"6c6128144aa620bfae7630e97fd32114706f3f724b42af7faa4cb7a7c554ab1e"}
Nov 02 18:16:02.188998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:02.188995    4456 scope.go:122] "RemoveContainer" containerID="d88d0bddc4d93ed42f5f2ffb943fc376a1de597ed4368f492c206bca841a6ef3"
Nov 02 18:16:02.200227 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:02.200205    4456 scope.go:122] "RemoveContainer" containerID="d88d0bddc4d93ed42f5f2ffb943fc376a1de597ed4368f492c206bca841a6ef3"
Nov 02 18:16:02.200510 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:16:02.200487    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"d88d0bddc4d93ed42f5f2ffb943fc376a1de597ed4368f492c206bca841a6ef3\": container with ID starting with d88d0bddc4d93ed42f5f2ffb943fc376a1de597ed4368f492c206bca841a6ef3 not found: ID does not exist" containerID="d88d0bddc4d93ed42f5f2ffb943fc376a1de597ed4368f492c206bca841a6ef3"
Nov 02 18:16:02.200578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:02.200522    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"d88d0bddc4d93ed42f5f2ffb943fc376a1de597ed4368f492c206bca841a6ef3"} err="failed to get container status \"d88d0bddc4d93ed42f5f2ffb943fc376a1de597ed4368f492c206bca841a6ef3\": rpc error: code = NotFound desc = could not find container \"d88d0bddc4d93ed42f5f2ffb943fc376a1de597ed4368f492c206bca841a6ef3\": container with ID starting with d88d0bddc4d93ed42f5f2ffb943fc376a1de597ed4368f492c206bca841a6ef3 not found: ID does not exist"
Nov 02 18:16:02.244614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:02.244592    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"]
Nov 02 18:16:02.247114 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:02.247089    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["e2e-models-e2e-worker-w0-069fa5/premium-w0-069fa5-kserve-699ddf764c-xmspr"]
Nov 02 18:16:03.612456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:03.612421    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" path="/var/lib/kubelet/pods/9e85cf3f-eb7a-452b-b716-26df5b391065/volumes"
Nov 02 18:16:10.158243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:10.158207    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:10.615444 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:10.615405    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:10.717957 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:10.717920    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"]
Nov 02 18:16:10.731206 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:10.731179    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-keys-e2e-worker-w0-069fa5-glgpf"]
Nov 02 18:16:10.763016 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:10.762979    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:11.262405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:11.262377    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-driver/0.log"
Nov 02 18:16:11.278424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:11.278394    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:11.307960 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:11.307934    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-driver/0.log"
Nov 02 18:16:11.614925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:11.614848    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9b3dd050-cbf9-4796-8440-1422c946dfa5" path="/var/lib/kubelet/pods/9b3dd050-cbf9-4796-8440-1422c946dfa5/volumes"
Nov 02 18:16:16.164809 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:16:16.164767    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:16:17.853648 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:17.853612    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:17.996288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:17.996248    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:18.222772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:18.222744    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:24.023727 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:24.023685    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:24.164680 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:24.164637    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:24.962413 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:24.962381    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:30.644993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:30.644954    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:38.499432 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:38.499394    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:46.165062 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:16:46.165019    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:16:47.847132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.847100    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"]
Nov 02 18:16:47.847519 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.847478    4456 cpu_manager.go:414] "containerMap: removing container" podUID="573c6b8d-87c3-486a-8a63-1b6b409103e7" containerName="revoke-keys"
Nov 02 18:16:47.847519 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.847488    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="573c6b8d-87c3-486a-8a63-1b6b409103e7" containerName="revoke-keys"
Nov 02 18:16:47.847519 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.847507    4456 cpu_manager.go:414] "containerMap: removing container" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main"
Nov 02 18:16:47.847519 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.847512    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main"
Nov 02 18:16:47.847654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.847527    4456 cpu_manager.go:414] "containerMap: removing container" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing"
Nov 02 18:16:47.847654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.847533    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing"
Nov 02 18:16:47.847654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.847609    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="573c6b8d-87c3-486a-8a63-1b6b409103e7" containerName="revoke-keys"
Nov 02 18:16:47.847654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.847619    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="9e85cf3f-eb7a-452b-b716-26df5b391065" containerName="main"
Nov 02 18:16:47.847654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.847627    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="b8847684-5676-4253-850f-9f2b2c42ea0c" containerName="payload-processing"
Nov 02 18:16:47.850751 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.850733    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.869010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.868983    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce08d5b-d93c-4c5f-bb03-89c460372a06-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.869157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.869021    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce08d5b-d93c-4c5f-bb03-89c460372a06-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.869157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.869041    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce08d5b-d93c-4c5f-bb03-89c460372a06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.869157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.869091    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce08d5b-d93c-4c5f-bb03-89c460372a06-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.869157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.869143    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-f5mlr\" (UniqueName: \"kubernetes.io/projected/cce08d5b-d93c-4c5f-bb03-89c460372a06-kube-api-access-f5mlr\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.869381 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.869278    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce08d5b-d93c-4c5f-bb03-89c460372a06-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.869381 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.869346    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce08d5b-d93c-4c5f-bb03-89c460372a06-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.869496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.869388    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce08d5b-d93c-4c5f-bb03-89c460372a06-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.869496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.869418    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce08d5b-d93c-4c5f-bb03-89c460372a06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.885106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.885076    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"]
Nov 02 18:16:47.970210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.970172    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce08d5b-d93c-4c5f-bb03-89c460372a06-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.970210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.970213    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce08d5b-d93c-4c5f-bb03-89c460372a06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.970449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.970232    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce08d5b-d93c-4c5f-bb03-89c460372a06-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.970449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.970255    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce08d5b-d93c-4c5f-bb03-89c460372a06-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.970449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.970271    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce08d5b-d93c-4c5f-bb03-89c460372a06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.970449 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.970318    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce08d5b-d93c-4c5f-bb03-89c460372a06-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.970596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.970342    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-f5mlr\" (UniqueName: \"kubernetes.io/projected/cce08d5b-d93c-4c5f-bb03-89c460372a06-kube-api-access-f5mlr\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.970596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.970381    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce08d5b-d93c-4c5f-bb03-89c460372a06-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.970596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.970444    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce08d5b-d93c-4c5f-bb03-89c460372a06-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.970701 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.970662    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/cce08d5b-d93c-4c5f-bb03-89c460372a06-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.971083 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.970988    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce08d5b-d93c-4c5f-bb03-89c460372a06-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.971083 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.971018    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/cce08d5b-d93c-4c5f-bb03-89c460372a06-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.971083 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.971072    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/cce08d5b-d93c-4c5f-bb03-89c460372a06-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.971083 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.971095    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/cce08d5b-d93c-4c5f-bb03-89c460372a06-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.972805 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.972781    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/cce08d5b-d93c-4c5f-bb03-89c460372a06-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.973040 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.973023    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/cce08d5b-d93c-4c5f-bb03-89c460372a06-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.994713 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.994691    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/cce08d5b-d93c-4c5f-bb03-89c460372a06-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:47.994858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:47.994772    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-f5mlr\" (UniqueName: \"kubernetes.io/projected/cce08d5b-d93c-4c5f-bb03-89c460372a06-kube-api-access-f5mlr\") pod \"maas-default-gateway-openshift-default-76f4876667-r5jnj\" (UID: \"cce08d5b-d93c-4c5f-bb03-89c460372a06\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:48.167752 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:48.167658    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:48.485990 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:48.485943    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"]
Nov 02 18:16:48.486523 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:16:48.486495    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podcce08d5b_d93c_4c5f_bb03_89c460372a06.slice/crio-3fe0f5bffc52e3df7da1bfd9eb15d302a4ef60263615342facd3c70a30ede46b WatchSource:0}: Error finding container 3fe0f5bffc52e3df7da1bfd9eb15d302a4ef60263615342facd3c70a30ede46b: Status 404 returned error can't find the container with id 3fe0f5bffc52e3df7da1bfd9eb15d302a4ef60263615342facd3c70a30ede46b
Nov 02 18:16:48.488169 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:48.488141    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:16:48.488242 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:48.488205    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:16:48.488280 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:48.488241    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:16:49.512757 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:49.512718    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj" event={"ID":"cce08d5b-d93c-4c5f-bb03-89c460372a06","Type":"ContainerStarted","Data":"7713cd43295b363dbd6bd76d80742d1d4a18935e9fbe3cd1c1707a996cf12e55"}
Nov 02 18:16:49.513270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:49.512779    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj" event={"ID":"cce08d5b-d93c-4c5f-bb03-89c460372a06","Type":"ContainerStarted","Data":"3fe0f5bffc52e3df7da1bfd9eb15d302a4ef60263615342facd3c70a30ede46b"}
Nov 02 18:16:49.574532 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:49.574477    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj" podStartSLOduration=2.574460393 podStartE2EDuration="2.574460393s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:16:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:16:49.573440782 +0000 UTC m=+2826.064503968" watchObservedRunningTime="2026-11-02 18:16:49.574460393 +0000 UTC m=+2826.065523549"
Nov 02 18:16:50.168626 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:50.168594    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:50.169881 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:50.169850    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-r5jnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 02 18:16:50.170017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:50.169913    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj" podUID="cce08d5b-d93c-4c5f-bb03-89c460372a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 02 18:16:51.168225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:51.168192    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-r5jnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 02 18:16:51.168632 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:51.168244    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj" podUID="cce08d5b-d93c-4c5f-bb03-89c460372a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 02 18:16:52.168446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:52.168407    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-r5jnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 02 18:16:52.168970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:52.168464    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj" podUID="cce08d5b-d93c-4c5f-bb03-89c460372a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 02 18:16:53.168368 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:53.168329    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-r5jnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 02 18:16:53.168540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:53.168385    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj" podUID="cce08d5b-d93c-4c5f-bb03-89c460372a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 02 18:16:54.168539 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:54.168501    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-r5jnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 02 18:16:54.168969 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:54.168582    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj" podUID="cce08d5b-d93c-4c5f-bb03-89c460372a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 02 18:16:55.168775 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:55.168740    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-r5jnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused" start-of-body=
Nov 02 18:16:55.169217 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:55.168799    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj" podUID="cce08d5b-d93c-4c5f-bb03-89c460372a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": dial tcp 10.134.1.26:15021: connect: connection refused"
Nov 02 18:16:57.169871 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:57.169809    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-r5jnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:16:57.170417 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:57.169895    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj" podUID="cce08d5b-d93c-4c5f-bb03-89c460372a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:16:58.212015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:58.211969    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-r5jnj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:16:58.212540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:58.212044    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj" podUID="cce08d5b-d93c-4c5f-bb03-89c460372a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:16:58.647239 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:58.647154    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:16:58.894966 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:58.894940    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:58.895393 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:58.895367    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:16:58.896030 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:16:58.896012    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj"
Nov 02 18:17:00.908783 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:00.908746    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:01.061878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:01.061843    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"]
Nov 02 18:17:01.075110 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:01.075071    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"]
Nov 02 18:17:01.077540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:01.077519    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6ac0b134-knqf5"]
Nov 02 18:17:01.079860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:01.079835    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-95f424e6-fzsg5"]
Nov 02 18:17:01.612612 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:01.612574    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="2a8cf6f4-2d7a-4b31-ae04-1192864b3610" path="/var/lib/kubelet/pods/2a8cf6f4-2d7a-4b31-ae04-1192864b3610/volumes"
Nov 02 18:17:01.613044 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:01.613029    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="930e728c-2ee0-4c90-aa3f-59ab9c88a012" path="/var/lib/kubelet/pods/930e728c-2ee0-4c90-aa3f-59ab9c88a012/volumes"
Nov 02 18:17:02.895828 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:02.895776    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"]
Nov 02 18:17:02.901151 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:02.901123    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:02.924413 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:02.924382    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"]
Nov 02 18:17:03.039628 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.039597    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d114747-c309-445d-8159-5e532bede175-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.039795 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.039633    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d114747-c309-445d-8159-5e532bede175-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.039795 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.039653    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d7gz5\" (UniqueName: \"kubernetes.io/projected/9d114747-c309-445d-8159-5e532bede175-kube-api-access-d7gz5\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.039795 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.039677    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d114747-c309-445d-8159-5e532bede175-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.039934 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.039798    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d114747-c309-445d-8159-5e532bede175-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.039934 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.039848    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d114747-c309-445d-8159-5e532bede175-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.039997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.039944    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d114747-c309-445d-8159-5e532bede175-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.040030 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.040006    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d114747-c309-445d-8159-5e532bede175-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.040065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.040035    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d114747-c309-445d-8159-5e532bede175-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.140652 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.140619    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d114747-c309-445d-8159-5e532bede175-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.140877 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.140680    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d114747-c309-445d-8159-5e532bede175-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.140877 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.140709    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d114747-c309-445d-8159-5e532bede175-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.140877 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.140732    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d7gz5\" (UniqueName: \"kubernetes.io/projected/9d114747-c309-445d-8159-5e532bede175-kube-api-access-d7gz5\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.140877 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.140763    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d114747-c309-445d-8159-5e532bede175-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.141127 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.140928    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d114747-c309-445d-8159-5e532bede175-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.141127 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.140968    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d114747-c309-445d-8159-5e532bede175-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.141127 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.141013    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d114747-c309-445d-8159-5e532bede175-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.141127 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.141029    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d114747-c309-445d-8159-5e532bede175-credential-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.141127 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.141121    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d114747-c309-445d-8159-5e532bede175-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.141387 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.141240    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/9d114747-c309-445d-8159-5e532bede175-workload-socket\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.141387 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.141319    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/9d114747-c309-445d-8159-5e532bede175-istio-data\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.141496 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.141440    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/9d114747-c309-445d-8159-5e532bede175-workload-certs\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.141551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.141519    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/9d114747-c309-445d-8159-5e532bede175-istiod-ca-cert\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.143352 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.143332    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/9d114747-c309-445d-8159-5e532bede175-istio-envoy\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.143485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.143470    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/9d114747-c309-445d-8159-5e532bede175-istio-podinfo\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.165639 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.165583    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d7gz5\" (UniqueName: \"kubernetes.io/projected/9d114747-c309-445d-8159-5e532bede175-kube-api-access-d7gz5\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.165639 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.165621    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/9d114747-c309-445d-8159-5e532bede175-istio-token\") pod \"maas-default-gateway-openshift-default-76f4876667-lrssd\" (UID: \"9d114747-c309-445d-8159-5e532bede175\") " pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.215284 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.215256    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:03.528591 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.528554    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"]
Nov 02 18:17:03.529043 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:17:03.529016    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9d114747_c309_445d_8159_5e532bede175.slice/crio-b01b5c46fe2b311aa8589ba19fae2fdc3bf914d4d53afd202ef3165fd4f1efdd WatchSource:0}: Error finding container b01b5c46fe2b311aa8589ba19fae2fdc3bf914d4d53afd202ef3165fd4f1efdd: Status 404 returned error can't find the container with id b01b5c46fe2b311aa8589ba19fae2fdc3bf914d4d53afd202ef3165fd4f1efdd
Nov 02 18:17:03.531252 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.531218    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:17:03.531343 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.531280    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:17:03.531343 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.531308    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:17:03.625021 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.624985    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd" event={"ID":"9d114747-c309-445d-8159-5e532bede175","Type":"ContainerStarted","Data":"0e823269c3bc0902a0f623406b081eb802fba6146ebb9df0ab2de6b2be73a1e8"}
Nov 02 18:17:03.625268 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.625042    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd" event={"ID":"9d114747-c309-445d-8159-5e532bede175","Type":"ContainerStarted","Data":"b01b5c46fe2b311aa8589ba19fae2fdc3bf914d4d53afd202ef3165fd4f1efdd"}
Nov 02 18:17:03.676551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:03.676494    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd" podStartSLOduration=1.67647922 podStartE2EDuration="1.67647922s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:17:02 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:17:03.674763259 +0000 UTC m=+2840.165826421" watchObservedRunningTime="2026-11-02 18:17:03.67647922 +0000 UTC m=+2840.167542380"
Nov 02 18:17:04.216272 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:04.216234    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:04.217718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:04.217696    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lrssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body=
Nov 02 18:17:04.217784 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:04.217747    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd" podUID="9d114747-c309-445d-8159-5e532bede175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused"
Nov 02 18:17:05.216238 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:05.216206    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lrssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body=
Nov 02 18:17:05.216424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:05.216258    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd" podUID="9d114747-c309-445d-8159-5e532bede175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused"
Nov 02 18:17:06.216010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:06.215964    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lrssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body=
Nov 02 18:17:06.216183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:06.216031    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd" podUID="9d114747-c309-445d-8159-5e532bede175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused"
Nov 02 18:17:07.216239 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:07.216207    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lrssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body=
Nov 02 18:17:07.216632 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:07.216262    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd" podUID="9d114747-c309-445d-8159-5e532bede175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused"
Nov 02 18:17:08.215700 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:08.215655    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lrssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body=
Nov 02 18:17:08.215934 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:08.215723    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd" podUID="9d114747-c309-445d-8159-5e532bede175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused"
Nov 02 18:17:09.216055 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:09.216017    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lrssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body=
Nov 02 18:17:09.216453 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:09.216075    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd" podUID="9d114747-c309-445d-8159-5e532bede175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused"
Nov 02 18:17:10.215628 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:10.215593    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lrssd container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused" start-of-body=
Nov 02 18:17:10.215800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:10.215644    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd" podUID="9d114747-c309-445d-8159-5e532bede175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": dial tcp 10.134.1.27:15021: connect: connection refused"
Nov 02 18:17:11.703731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:11.703641    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:11.704266 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:11.704243    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:11.705244 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:11.705219    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd"
Nov 02 18:17:13.427743 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:13.427711    4456 scope.go:122] "RemoveContainer" containerID="3736c6b20b77a62af51ed11e3d510954fe7f012d5fa027eb244cb94b608dd9cb"
Nov 02 18:17:16.164730 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:17:16.164689    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:17:34.031654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:34.031612    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"]
Nov 02 18:17:34.052505 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:34.052463    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"]
Nov 02 18:17:34.052683 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:34.052562    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-4fd065a0-2dxdl"]
Nov 02 18:17:34.055229 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:34.055204    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5cf75e7b-hk7n6"]
Nov 02 18:17:35.613084 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:35.613051    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="aed882b3-d3b9-49c4-8bac-ef4aa5004535" path="/var/lib/kubelet/pods/aed882b3-d3b9-49c4-8bac-ef4aa5004535/volumes"
Nov 02 18:17:35.613483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:35.613417    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5" path="/var/lib/kubelet/pods/fc4b930c-79d7-4beb-a5b0-0592c3bdcdd5/volumes"
Nov 02 18:17:38.637210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:38.637151    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:42.887168 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:42.887128    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:17:46.164424 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:17:46.164386    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:17:47.939004 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:47.938974    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"]
Nov 02 18:17:47.939519 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:47.939242    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj" podUID="3ffa306f-b84b-4bb9-ba77-e0bdaf58b777" containerName="discovery" containerID="cri-o://09bb8945087fb15758d5d9efebc5a08418929a47a699e984e7be398d49c03d69" gracePeriod=30
Nov 02 18:17:48.190883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.190825    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:17:48.332546 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.332512    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-csr-dns-cert\") pod \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") "
Nov 02 18:17:48.332546 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.332551    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-cacerts\" (UniqueName: \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-cacerts\") pod \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") "
Nov 02 18:17:48.332787 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.332572    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-local-certs\") pod \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") "
Nov 02 18:17:48.332787 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.332608    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-kube-api-access-ppv2m\" (UniqueName: \"kubernetes.io/projected/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-kube-api-access-ppv2m\") pod \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") "
Nov 02 18:17:48.332787 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.332629    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-csr-ca-configmap\") pod \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") "
Nov 02 18:17:48.332787 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.332694    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-kubeconfig\") pod \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") "
Nov 02 18:17:48.332787 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.332781    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-token\" (UniqueName: \"kubernetes.io/projected/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-token\") pod \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\" (UID: \"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777\") "
Nov 02 18:17:48.333195 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.333160    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-csr-ca-configmap" pod "3ffa306f-b84b-4bb9-ba77-e0bdaf58b777" (UID: "3ffa306f-b84b-4bb9-ba77-e0bdaf58b777"). InnerVolumeSpecName "istio-csr-ca-configmap". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:17:48.335543 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.335512    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-local-certs" pod "3ffa306f-b84b-4bb9-ba77-e0bdaf58b777" (UID: "3ffa306f-b84b-4bb9-ba77-e0bdaf58b777"). InnerVolumeSpecName "local-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:17:48.335543 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.335526    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-cacerts" pod "3ffa306f-b84b-4bb9-ba77-e0bdaf58b777" (UID: "3ffa306f-b84b-4bb9-ba77-e0bdaf58b777"). InnerVolumeSpecName "cacerts". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:17:48.335718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.335696    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-csr-dns-cert" pod "3ffa306f-b84b-4bb9-ba77-e0bdaf58b777" (UID: "3ffa306f-b84b-4bb9-ba77-e0bdaf58b777"). InnerVolumeSpecName "istio-csr-dns-cert". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:17:48.335768 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.335749    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-token" pod "3ffa306f-b84b-4bb9-ba77-e0bdaf58b777" (UID: "3ffa306f-b84b-4bb9-ba77-e0bdaf58b777"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:48.335804 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.335757    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-kubeconfig" pod "3ffa306f-b84b-4bb9-ba77-e0bdaf58b777" (UID: "3ffa306f-b84b-4bb9-ba77-e0bdaf58b777"). InnerVolumeSpecName "istio-kubeconfig". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:17:48.335870 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.335800    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-kube-api-access-ppv2m" pod "3ffa306f-b84b-4bb9-ba77-e0bdaf58b777" (UID: "3ffa306f-b84b-4bb9-ba77-e0bdaf58b777"). InnerVolumeSpecName "kube-api-access-ppv2m". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:17:48.434410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.434372    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:17:48.434410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.434404    4456 reconciler_common.go:299] "Volume detached for volume \"istio-csr-dns-cert\" (UniqueName: \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-csr-dns-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:17:48.434410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.434416    4456 reconciler_common.go:299] "Volume detached for volume \"cacerts\" (UniqueName: \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-cacerts\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:17:48.434645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.434425    4456 reconciler_common.go:299] "Volume detached for volume \"local-certs\" (UniqueName: \"kubernetes.io/empty-dir/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-local-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:17:48.434645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.434435    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-ppv2m\" (UniqueName: \"kubernetes.io/projected/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-kube-api-access-ppv2m\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:17:48.434645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.434445    4456 reconciler_common.go:299] "Volume detached for volume \"istio-csr-ca-configmap\" (UniqueName: \"kubernetes.io/configmap/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-csr-ca-configmap\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:17:48.434645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.434453    4456 reconciler_common.go:299] "Volume detached for volume \"istio-kubeconfig\" (UniqueName: \"kubernetes.io/secret/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777-istio-kubeconfig\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:17:48.910001 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.909964    4456 generic.go:356] "Generic (PLEG): container finished" podID="3ffa306f-b84b-4bb9-ba77-e0bdaf58b777" containerID="09bb8945087fb15758d5d9efebc5a08418929a47a699e984e7be398d49c03d69" exitCode=0
Nov 02 18:17:48.910183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.910011    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"
Nov 02 18:17:48.910183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.910011    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj" event={"ID":"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777","Type":"ContainerDied","Data":"09bb8945087fb15758d5d9efebc5a08418929a47a699e984e7be398d49c03d69"}
Nov 02 18:17:48.910183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.910125    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj" event={"ID":"3ffa306f-b84b-4bb9-ba77-e0bdaf58b777","Type":"ContainerDied","Data":"b32c466b30fabb8ae5aabb7783a3233427c97e8dff62c70d6f8f394e4408d4c0"}
Nov 02 18:17:48.910183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.910145    4456 scope.go:122] "RemoveContainer" containerID="09bb8945087fb15758d5d9efebc5a08418929a47a699e984e7be398d49c03d69"
Nov 02 18:17:48.921000 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.920975    4456 scope.go:122] "RemoveContainer" containerID="09bb8945087fb15758d5d9efebc5a08418929a47a699e984e7be398d49c03d69"
Nov 02 18:17:48.921278 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:17:48.921260    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09bb8945087fb15758d5d9efebc5a08418929a47a699e984e7be398d49c03d69\": container with ID starting with 09bb8945087fb15758d5d9efebc5a08418929a47a699e984e7be398d49c03d69 not found: ID does not exist" containerID="09bb8945087fb15758d5d9efebc5a08418929a47a699e984e7be398d49c03d69"
Nov 02 18:17:48.921331 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.921286    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09bb8945087fb15758d5d9efebc5a08418929a47a699e984e7be398d49c03d69"} err="failed to get container status \"09bb8945087fb15758d5d9efebc5a08418929a47a699e984e7be398d49c03d69\": rpc error: code = NotFound desc = could not find container \"09bb8945087fb15758d5d9efebc5a08418929a47a699e984e7be398d49c03d69\": container with ID starting with 09bb8945087fb15758d5d9efebc5a08418929a47a699e984e7be398d49c03d69 not found: ID does not exist"
Nov 02 18:17:48.981114 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.981087    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"]
Nov 02 18:17:48.994299 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:48.994276    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/istiod-openshift-gateway-7ddd9cfdc8-6bvgj"]
Nov 02 18:17:49.612505 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:17:49.612472    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="3ffa306f-b84b-4bb9-ba77-e0bdaf58b777" path="/var/lib/kubelet/pods/3ffa306f-b84b-4bb9-ba77-e0bdaf58b777/volumes"
Nov 02 18:18:09.639985 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.639933    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"]
Nov 02 18:18:09.640437 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.640336    4456 cpu_manager.go:414] "containerMap: removing container" podUID="3ffa306f-b84b-4bb9-ba77-e0bdaf58b777" containerName="discovery"
Nov 02 18:18:09.640437 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.640347    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="3ffa306f-b84b-4bb9-ba77-e0bdaf58b777" containerName="discovery"
Nov 02 18:18:09.640437 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.640429    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="3ffa306f-b84b-4bb9-ba77-e0bdaf58b777" containerName="discovery"
Nov 02 18:18:09.644896 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.644875    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"
Nov 02 18:18:09.663098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.663067    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"]
Nov 02 18:18:09.673729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.673702    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:18:09.673882 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.673762    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:18:09.674065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.674047    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:18:09.674110 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.674037    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:18:09.682103 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.682074    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7584f343-9a50-499d-a96b-132079baeff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2gk4j\" (UID: \"7584f343-9a50-499d-a96b-132079baeff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"
Nov 02 18:18:09.682216 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.682201    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-h2p7n\" (UniqueName: \"kubernetes.io/projected/7584f343-9a50-499d-a96b-132079baeff7-kube-api-access-h2p7n\") pod \"maas-api-revoke-sub-2e1f5b5f-2gk4j\" (UID: \"7584f343-9a50-499d-a96b-132079baeff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"
Nov 02 18:18:09.682260 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.682231    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7584f343-9a50-499d-a96b-132079baeff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2gk4j\" (UID: \"7584f343-9a50-499d-a96b-132079baeff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"
Nov 02 18:18:09.782838 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.782789    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-h2p7n\" (UniqueName: \"kubernetes.io/projected/7584f343-9a50-499d-a96b-132079baeff7-kube-api-access-h2p7n\") pod \"maas-api-revoke-sub-2e1f5b5f-2gk4j\" (UID: \"7584f343-9a50-499d-a96b-132079baeff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"
Nov 02 18:18:09.783013 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.782852    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7584f343-9a50-499d-a96b-132079baeff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2gk4j\" (UID: \"7584f343-9a50-499d-a96b-132079baeff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"
Nov 02 18:18:09.783013 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.782922    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7584f343-9a50-499d-a96b-132079baeff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2gk4j\" (UID: \"7584f343-9a50-499d-a96b-132079baeff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"
Nov 02 18:18:09.783495 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.783473    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7584f343-9a50-499d-a96b-132079baeff7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2gk4j\" (UID: \"7584f343-9a50-499d-a96b-132079baeff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"
Nov 02 18:18:09.785330 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.785309    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7584f343-9a50-499d-a96b-132079baeff7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-2e1f5b5f-2gk4j\" (UID: \"7584f343-9a50-499d-a96b-132079baeff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"
Nov 02 18:18:09.807332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.807304    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-h2p7n\" (UniqueName: \"kubernetes.io/projected/7584f343-9a50-499d-a96b-132079baeff7-kube-api-access-h2p7n\") pod \"maas-api-revoke-sub-2e1f5b5f-2gk4j\" (UID: \"7584f343-9a50-499d-a96b-132079baeff7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"
Nov 02 18:18:09.958428 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:09.958400    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"
Nov 02 18:18:10.258078 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:10.258047    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"]
Nov 02 18:18:10.258455 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:18:10.258433    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7584f343_9a50_499d_a96b_132079baeff7.slice/crio-f3fbcd47e02d3ff110c633e161481e767a0389c0d608d7c375e4c782176bb3a9 WatchSource:0}: Error finding container f3fbcd47e02d3ff110c633e161481e767a0389c0d608d7c375e4c782176bb3a9: Status 404 returned error can't find the container with id f3fbcd47e02d3ff110c633e161481e767a0389c0d608d7c375e4c782176bb3a9
Nov 02 18:18:11.068884 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:11.068846    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j" event={"ID":"7584f343-9a50-499d-a96b-132079baeff7","Type":"ContainerStarted","Data":"aabae281835c289c50c96a5e221bc132e31fb1da3a3358cdcf310c506d5aff37"}
Nov 02 18:18:11.069279 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:11.068908    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j" event={"ID":"7584f343-9a50-499d-a96b-132079baeff7","Type":"ContainerStarted","Data":"f3fbcd47e02d3ff110c633e161481e767a0389c0d608d7c375e4c782176bb3a9"}
Nov 02 18:18:11.112061 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:11.112019    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j" podStartSLOduration=2.112005019 podStartE2EDuration="2.112005019s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:09 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:18:11.111312382 +0000 UTC m=+2907.602375545" watchObservedRunningTime="2026-11-02 18:18:11.112005019 +0000 UTC m=+2907.603068180"
Nov 02 18:18:12.076408 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:12.076371    4456 generic.go:356] "Generic (PLEG): container finished" podID="7584f343-9a50-499d-a96b-132079baeff7" containerID="aabae281835c289c50c96a5e221bc132e31fb1da3a3358cdcf310c506d5aff37" exitCode=0
Nov 02 18:18:12.076867 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:12.076476    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j" event={"ID":"7584f343-9a50-499d-a96b-132079baeff7","Type":"ContainerDied","Data":"aabae281835c289c50c96a5e221bc132e31fb1da3a3358cdcf310c506d5aff37"}
Nov 02 18:18:13.217521 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.217498    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"
Nov 02 18:18:13.309182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.309154    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7584f343-9a50-499d-a96b-132079baeff7-kube-api-access-h2p7n\" (UniqueName: \"kubernetes.io/projected/7584f343-9a50-499d-a96b-132079baeff7-kube-api-access-h2p7n\") pod \"7584f343-9a50-499d-a96b-132079baeff7\" (UID: \"7584f343-9a50-499d-a96b-132079baeff7\") "
Nov 02 18:18:13.309344 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.309249    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7584f343-9a50-499d-a96b-132079baeff7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7584f343-9a50-499d-a96b-132079baeff7-maas-api-openshift-ca\") pod \"7584f343-9a50-499d-a96b-132079baeff7\" (UID: \"7584f343-9a50-499d-a96b-132079baeff7\") "
Nov 02 18:18:13.309344 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.309283    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7584f343-9a50-499d-a96b-132079baeff7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7584f343-9a50-499d-a96b-132079baeff7-maas-api-xks-ca\") pod \"7584f343-9a50-499d-a96b-132079baeff7\" (UID: \"7584f343-9a50-499d-a96b-132079baeff7\") "
Nov 02 18:18:13.309604 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.309585    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7584f343-9a50-499d-a96b-132079baeff7-maas-api-openshift-ca" pod "7584f343-9a50-499d-a96b-132079baeff7" (UID: "7584f343-9a50-499d-a96b-132079baeff7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:18:13.311373 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.311351    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7584f343-9a50-499d-a96b-132079baeff7-kube-api-access-h2p7n" pod "7584f343-9a50-499d-a96b-132079baeff7" (UID: "7584f343-9a50-499d-a96b-132079baeff7"). InnerVolumeSpecName "kube-api-access-h2p7n". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:18:13.311530 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.311511    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7584f343-9a50-499d-a96b-132079baeff7-maas-api-xks-ca" pod "7584f343-9a50-499d-a96b-132079baeff7" (UID: "7584f343-9a50-499d-a96b-132079baeff7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:18:13.410098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.410027    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/7584f343-9a50-499d-a96b-132079baeff7-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:18:13.410098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.410053    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/7584f343-9a50-499d-a96b-132079baeff7-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:18:13.410098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.410063    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-h2p7n\" (UniqueName: \"kubernetes.io/projected/7584f343-9a50-499d-a96b-132079baeff7-kube-api-access-h2p7n\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:18:13.456355 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.456325    4456 scope.go:122] "RemoveContainer" containerID="9867f9993570e8f274c2e9b7b82271760e52385bdbb74f70bc0aa8a29f039774"
Nov 02 18:18:13.466309 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.466290    4456 scope.go:122] "RemoveContainer" containerID="9fd4431a4f8d70b237e6855e1443cdcf43cce8d0e7028815878dab823ab38898"
Nov 02 18:18:13.474673 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.474653    4456 scope.go:122] "RemoveContainer" containerID="8edb03650dca6b1935c892249b69bc92c07709952f03b11a0207dba795beaa1c"
Nov 02 18:18:13.483260 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.483237    4456 scope.go:122] "RemoveContainer" containerID="0312f401b957bf64103e1e2ec02100ed3dd682d64ce32741d12e521889093217"
Nov 02 18:18:13.493874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.493852    4456 scope.go:122] "RemoveContainer" containerID="852b8f642a03ab7a55b5e40b2fec8845a457e699441ea49ae7a5e4e8e30b2e12"
Nov 02 18:18:13.502107 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.502092    4456 scope.go:122] "RemoveContainer" containerID="fca79505dcc07d2a45ecf9c33d89ff089634354e7540cedb2dca53cbb393a362"
Nov 02 18:18:13.510387 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.510363    4456 scope.go:122] "RemoveContainer" containerID="f8d3fa499482e951efc1422c6ad8a74b9e54eb1a76cdf63f582ca889d854e091"
Nov 02 18:18:13.518560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.518544    4456 scope.go:122] "RemoveContainer" containerID="92cdf4068fea46ac976b13d8439a63750fbece3874b0e28b3203b8dac5d5fec8"
Nov 02 18:18:13.526732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.526709    4456 scope.go:122] "RemoveContainer" containerID="0d54b34fabde9aeebc7f496fae68a4cd7e94840f60210fe2e1bf0a6697ed7dfd"
Nov 02 18:18:13.534656 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.534631    4456 scope.go:122] "RemoveContainer" containerID="c62f25f4e3ed9c7ecded4a01aec626d0b95966058426c9da278a522fb99b9280"
Nov 02 18:18:13.542358 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.542342    4456 scope.go:122] "RemoveContainer" containerID="3950d86680a5cef109cf16b5f6c6460db4ed523ab353de3a891d0a896661c2ac"
Nov 02 18:18:13.550270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.550252    4456 scope.go:122] "RemoveContainer" containerID="0b5be95732ef78cdc1af5066503c626a4c9cb8499de4dbb50308f359c6358fc4"
Nov 02 18:18:13.558185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.558163    4456 scope.go:122] "RemoveContainer" containerID="1005e3ad004e76a6d733579931a774a5011da75161fb0c4938ed55c9328c5248"
Nov 02 18:18:13.567445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:13.567429    4456 scope.go:122] "RemoveContainer" containerID="fb25944faff8de31645896efa975646aa529d38504de0a61e50895077dfb97f1"
Nov 02 18:18:14.091412 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:14.091382    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"
Nov 02 18:18:14.091608 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:14.091421    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j" event={"ID":"7584f343-9a50-499d-a96b-132079baeff7","Type":"ContainerDied","Data":"f3fbcd47e02d3ff110c633e161481e767a0389c0d608d7c375e4c782176bb3a9"}
Nov 02 18:18:14.091608 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:14.091473    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f3fbcd47e02d3ff110c633e161481e767a0389c0d608d7c375e4c782176bb3a9"
Nov 02 18:18:16.164282 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:18:16.164245    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:18:21.054055 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.054019    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"]
Nov 02 18:18:21.054653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.054632    4456 cpu_manager.go:414] "containerMap: removing container" podUID="7584f343-9a50-499d-a96b-132079baeff7" containerName="revoke-keys"
Nov 02 18:18:21.054736 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.054655    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7584f343-9a50-499d-a96b-132079baeff7" containerName="revoke-keys"
Nov 02 18:18:21.054809 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.054796    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7584f343-9a50-499d-a96b-132079baeff7" containerName="revoke-keys"
Nov 02 18:18:21.058131 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.058109    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"
Nov 02 18:18:21.077066 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.077030    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"]
Nov 02 18:18:21.079514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.079484    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c20671-406b-4e7a-80fe-6d4a1b155980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-72gr8\" (UID: \"88c20671-406b-4e7a-80fe-6d4a1b155980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"
Nov 02 18:18:21.079656 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.079533    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c20671-406b-4e7a-80fe-6d4a1b155980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-72gr8\" (UID: \"88c20671-406b-4e7a-80fe-6d4a1b155980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"
Nov 02 18:18:21.079720 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.079675    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-gw5vm\" (UniqueName: \"kubernetes.io/projected/88c20671-406b-4e7a-80fe-6d4a1b155980-kube-api-access-gw5vm\") pod \"maas-api-revoke-sub-ec843018-72gr8\" (UID: \"88c20671-406b-4e7a-80fe-6d4a1b155980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"
Nov 02 18:18:21.086590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.086563    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:18:21.086590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.086606    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:18:21.086841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.086629    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:18:21.087630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.087608    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:18:21.180860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.180799    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-gw5vm\" (UniqueName: \"kubernetes.io/projected/88c20671-406b-4e7a-80fe-6d4a1b155980-kube-api-access-gw5vm\") pod \"maas-api-revoke-sub-ec843018-72gr8\" (UID: \"88c20671-406b-4e7a-80fe-6d4a1b155980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"
Nov 02 18:18:21.181058 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.180911    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c20671-406b-4e7a-80fe-6d4a1b155980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-72gr8\" (UID: \"88c20671-406b-4e7a-80fe-6d4a1b155980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"
Nov 02 18:18:21.181058 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.180951    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c20671-406b-4e7a-80fe-6d4a1b155980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-72gr8\" (UID: \"88c20671-406b-4e7a-80fe-6d4a1b155980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"
Nov 02 18:18:21.181560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.181533    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c20671-406b-4e7a-80fe-6d4a1b155980-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ec843018-72gr8\" (UID: \"88c20671-406b-4e7a-80fe-6d4a1b155980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"
Nov 02 18:18:21.183535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.183513    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c20671-406b-4e7a-80fe-6d4a1b155980-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ec843018-72gr8\" (UID: \"88c20671-406b-4e7a-80fe-6d4a1b155980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"
Nov 02 18:18:21.206022 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.205998    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-gw5vm\" (UniqueName: \"kubernetes.io/projected/88c20671-406b-4e7a-80fe-6d4a1b155980-kube-api-access-gw5vm\") pod \"maas-api-revoke-sub-ec843018-72gr8\" (UID: \"88c20671-406b-4e7a-80fe-6d4a1b155980\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"
Nov 02 18:18:21.371948 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.371857    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"
Nov 02 18:18:21.677301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:21.677270    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"]
Nov 02 18:18:21.677967 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:18:21.677946    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod88c20671_406b_4e7a_80fe_6d4a1b155980.slice/crio-f2261a31accc6d5aa6f3708f76d8fff9ecf595181c01a65a582b061a6e8f60f2 WatchSource:0}: Error finding container f2261a31accc6d5aa6f3708f76d8fff9ecf595181c01a65a582b061a6e8f60f2: Status 404 returned error can't find the container with id f2261a31accc6d5aa6f3708f76d8fff9ecf595181c01a65a582b061a6e8f60f2
Nov 02 18:18:22.139583 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:22.139543    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8" event={"ID":"88c20671-406b-4e7a-80fe-6d4a1b155980","Type":"ContainerStarted","Data":"82a39ba47d98ca26c5525d06cb177da7d0e1c2cc2bba443b377e3ed46d18afc5"}
Nov 02 18:18:22.139982 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:22.139608    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8" event={"ID":"88c20671-406b-4e7a-80fe-6d4a1b155980","Type":"ContainerStarted","Data":"f2261a31accc6d5aa6f3708f76d8fff9ecf595181c01a65a582b061a6e8f60f2"}
Nov 02 18:18:22.183334 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:22.183277    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8" podStartSLOduration=1.183262554 podStartE2EDuration="1.183262554s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:18:22.181863917 +0000 UTC m=+2918.672927093" watchObservedRunningTime="2026-11-02 18:18:22.183262554 +0000 UTC m=+2918.674325711"
Nov 02 18:18:23.031207 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:23.031170    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"]
Nov 02 18:18:23.044150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:23.044118    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"]
Nov 02 18:18:23.046365 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:23.046346    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-bf7cfcae-bnbxx"]
Nov 02 18:18:23.048387 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:23.048360    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-be9a29be-6qzx5"]
Nov 02 18:18:23.147240 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:23.147207    4456 generic.go:356] "Generic (PLEG): container finished" podID="88c20671-406b-4e7a-80fe-6d4a1b155980" containerID="82a39ba47d98ca26c5525d06cb177da7d0e1c2cc2bba443b377e3ed46d18afc5" exitCode=0
Nov 02 18:18:23.147603 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:23.147297    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8" event={"ID":"88c20671-406b-4e7a-80fe-6d4a1b155980","Type":"ContainerDied","Data":"82a39ba47d98ca26c5525d06cb177da7d0e1c2cc2bba443b377e3ed46d18afc5"}
Nov 02 18:18:23.612520 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:23.612487    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ac13cb59-6208-4268-818f-205e6105789b" path="/var/lib/kubelet/pods/ac13cb59-6208-4268-818f-205e6105789b/volumes"
Nov 02 18:18:23.612906 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:23.612892    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="cc0bc138-76e3-41ca-83ac-c32c5a6770ee" path="/var/lib/kubelet/pods/cc0bc138-76e3-41ca-83ac-c32c5a6770ee/volumes"
Nov 02 18:18:24.294055 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:24.294026    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"
Nov 02 18:18:24.418463 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:24.418378    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/88c20671-406b-4e7a-80fe-6d4a1b155980-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c20671-406b-4e7a-80fe-6d4a1b155980-maas-api-xks-ca\") pod \"88c20671-406b-4e7a-80fe-6d4a1b155980\" (UID: \"88c20671-406b-4e7a-80fe-6d4a1b155980\") "
Nov 02 18:18:24.418629 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:24.418502    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/88c20671-406b-4e7a-80fe-6d4a1b155980-kube-api-access-gw5vm\" (UniqueName: \"kubernetes.io/projected/88c20671-406b-4e7a-80fe-6d4a1b155980-kube-api-access-gw5vm\") pod \"88c20671-406b-4e7a-80fe-6d4a1b155980\" (UID: \"88c20671-406b-4e7a-80fe-6d4a1b155980\") "
Nov 02 18:18:24.418629 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:24.418553    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/88c20671-406b-4e7a-80fe-6d4a1b155980-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c20671-406b-4e7a-80fe-6d4a1b155980-maas-api-openshift-ca\") pod \"88c20671-406b-4e7a-80fe-6d4a1b155980\" (UID: \"88c20671-406b-4e7a-80fe-6d4a1b155980\") "
Nov 02 18:18:24.419018 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:24.418984    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/88c20671-406b-4e7a-80fe-6d4a1b155980-maas-api-openshift-ca" pod "88c20671-406b-4e7a-80fe-6d4a1b155980" (UID: "88c20671-406b-4e7a-80fe-6d4a1b155980"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:18:24.420879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:24.420850    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/88c20671-406b-4e7a-80fe-6d4a1b155980-maas-api-xks-ca" pod "88c20671-406b-4e7a-80fe-6d4a1b155980" (UID: "88c20671-406b-4e7a-80fe-6d4a1b155980"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:18:24.420958 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:24.420919    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/88c20671-406b-4e7a-80fe-6d4a1b155980-kube-api-access-gw5vm" pod "88c20671-406b-4e7a-80fe-6d4a1b155980" (UID: "88c20671-406b-4e7a-80fe-6d4a1b155980"). InnerVolumeSpecName "kube-api-access-gw5vm". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:18:24.519480 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:24.519444    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-gw5vm\" (UniqueName: \"kubernetes.io/projected/88c20671-406b-4e7a-80fe-6d4a1b155980-kube-api-access-gw5vm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:18:24.519480 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:24.519473    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/88c20671-406b-4e7a-80fe-6d4a1b155980-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:18:24.519480 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:24.519484    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/88c20671-406b-4e7a-80fe-6d4a1b155980-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:18:25.033092 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:25.033054    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"]
Nov 02 18:18:25.046452 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:25.046398    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"]
Nov 02 18:18:25.048680 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:25.048655    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-cd103858-cjlgg"]
Nov 02 18:18:25.050730 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:25.050705    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e0b090ef-7b5wk"]
Nov 02 18:18:25.160993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:25.160959    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8" event={"ID":"88c20671-406b-4e7a-80fe-6d4a1b155980","Type":"ContainerDied","Data":"f2261a31accc6d5aa6f3708f76d8fff9ecf595181c01a65a582b061a6e8f60f2"}
Nov 02 18:18:25.161146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:25.161012    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f2261a31accc6d5aa6f3708f76d8fff9ecf595181c01a65a582b061a6e8f60f2"
Nov 02 18:18:25.161146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:25.160981    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"
Nov 02 18:18:25.612548 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:25.612512    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="18e153e5-bd9d-4a00-8c94-75f6a53af87c" path="/var/lib/kubelet/pods/18e153e5-bd9d-4a00-8c94-75f6a53af87c/volumes"
Nov 02 18:18:25.612945 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:25.612927    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="887b756a-8d5e-4510-9f48-0223365a0543" path="/var/lib/kubelet/pods/887b756a-8d5e-4510-9f48-0223365a0543/volumes"
Nov 02 18:18:34.031924 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:34.031880    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"]
Nov 02 18:18:34.044872 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:34.044843    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3af94642-hjdmc"]
Nov 02 18:18:34.047251 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:34.047227    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"]
Nov 02 18:18:34.049414 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:34.049395    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"]
Nov 02 18:18:34.051352 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:34.051329    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"]
Nov 02 18:18:34.057610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:34.057592    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-1871ff23-sthwn"]
Nov 02 18:18:34.059673 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:34.059654    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-6bf0d7a5-dk2dp"]
Nov 02 18:18:34.061384 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:34.061367    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e426cc5f-pp2qt"]
Nov 02 18:18:35.612093 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:35.612061    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="1d81d958-89bb-4f35-8c6c-d7a09d2507ae" path="/var/lib/kubelet/pods/1d81d958-89bb-4f35-8c6c-d7a09d2507ae/volumes"
Nov 02 18:18:35.612490 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:35.612476    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6cc67f3e-e8e3-4d57-859c-abb8561601a0" path="/var/lib/kubelet/pods/6cc67f3e-e8e3-4d57-859c-abb8561601a0/volumes"
Nov 02 18:18:35.612888 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:35.612874    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7e85b538-1702-463b-adc2-23276a23367e" path="/var/lib/kubelet/pods/7e85b538-1702-463b-adc2-23276a23367e/volumes"
Nov 02 18:18:35.613557 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:35.613544    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d95809ee-3d8e-4197-9792-342b1c7a4820" path="/var/lib/kubelet/pods/d95809ee-3d8e-4197-9792-342b1c7a4820/volumes"
Nov 02 18:18:38.792842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:38.791466    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:46.163988 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:18:46.163902    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:18:47.937712 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:47.937676    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"]
Nov 02 18:18:47.938381 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:47.938359    4456 cpu_manager.go:414] "containerMap: removing container" podUID="88c20671-406b-4e7a-80fe-6d4a1b155980" containerName="revoke-keys"
Nov 02 18:18:47.938435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:47.938385    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="88c20671-406b-4e7a-80fe-6d4a1b155980" containerName="revoke-keys"
Nov 02 18:18:47.938507 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:47.938496    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="88c20671-406b-4e7a-80fe-6d4a1b155980" containerName="revoke-keys"
Nov 02 18:18:47.941798 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:47.941783    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"
Nov 02 18:18:47.960156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:47.960132    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"]
Nov 02 18:18:47.970395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:47.970369    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:18:47.970395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:47.970374    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:18:47.970610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:47.970410    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:18:47.971237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:47.971224    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:18:48.068469 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:48.068425    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g627f\" (UniqueName: \"kubernetes.io/projected/73cfd712-6978-4e5b-a5a2-1641533d46d4-kube-api-access-g627f\") pod \"maas-api-revoke-sub-046e1343-kz6jz\" (UID: \"73cfd712-6978-4e5b-a5a2-1641533d46d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"
Nov 02 18:18:48.068669 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:48.068480    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73cfd712-6978-4e5b-a5a2-1641533d46d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-kz6jz\" (UID: \"73cfd712-6978-4e5b-a5a2-1641533d46d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"
Nov 02 18:18:48.068669 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:48.068649    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73cfd712-6978-4e5b-a5a2-1641533d46d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-kz6jz\" (UID: \"73cfd712-6978-4e5b-a5a2-1641533d46d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"
Nov 02 18:18:48.169600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:48.169562    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73cfd712-6978-4e5b-a5a2-1641533d46d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-kz6jz\" (UID: \"73cfd712-6978-4e5b-a5a2-1641533d46d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"
Nov 02 18:18:48.169797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:48.169639    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g627f\" (UniqueName: \"kubernetes.io/projected/73cfd712-6978-4e5b-a5a2-1641533d46d4-kube-api-access-g627f\") pod \"maas-api-revoke-sub-046e1343-kz6jz\" (UID: \"73cfd712-6978-4e5b-a5a2-1641533d46d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"
Nov 02 18:18:48.169797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:48.169676    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73cfd712-6978-4e5b-a5a2-1641533d46d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-kz6jz\" (UID: \"73cfd712-6978-4e5b-a5a2-1641533d46d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"
Nov 02 18:18:48.170361 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:48.170339    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73cfd712-6978-4e5b-a5a2-1641533d46d4-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-046e1343-kz6jz\" (UID: \"73cfd712-6978-4e5b-a5a2-1641533d46d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"
Nov 02 18:18:48.172101 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:48.172084    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73cfd712-6978-4e5b-a5a2-1641533d46d4-maas-api-xks-ca\") pod \"maas-api-revoke-sub-046e1343-kz6jz\" (UID: \"73cfd712-6978-4e5b-a5a2-1641533d46d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"
Nov 02 18:18:48.194508 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:48.194454    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g627f\" (UniqueName: \"kubernetes.io/projected/73cfd712-6978-4e5b-a5a2-1641533d46d4-kube-api-access-g627f\") pod \"maas-api-revoke-sub-046e1343-kz6jz\" (UID: \"73cfd712-6978-4e5b-a5a2-1641533d46d4\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"
Nov 02 18:18:48.255101 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:48.255063    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"
Nov 02 18:18:48.553546 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:48.553522    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"]
Nov 02 18:18:48.554302 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:18:48.554276    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod73cfd712_6978_4e5b_a5a2_1641533d46d4.slice/crio-2e649eca9af7f8145f16bf963ecb39151745eb6b90af931657f0c1a8ee171e7c WatchSource:0}: Error finding container 2e649eca9af7f8145f16bf963ecb39151745eb6b90af931657f0c1a8ee171e7c: Status 404 returned error can't find the container with id 2e649eca9af7f8145f16bf963ecb39151745eb6b90af931657f0c1a8ee171e7c
Nov 02 18:18:49.325512 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:49.325470    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz" event={"ID":"73cfd712-6978-4e5b-a5a2-1641533d46d4","Type":"ContainerStarted","Data":"8d3998bd52036ca16be6d5cd29056cdd91c2e28594d07f9b89ebce19e09bba4c"}
Nov 02 18:18:49.325956 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:49.325526    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz" event={"ID":"73cfd712-6978-4e5b-a5a2-1641533d46d4","Type":"ContainerStarted","Data":"2e649eca9af7f8145f16bf963ecb39151745eb6b90af931657f0c1a8ee171e7c"}
Nov 02 18:18:49.370130 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:49.370084    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz" podStartSLOduration=2.370069699 podStartE2EDuration="2.370069699s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:18:47 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:18:49.36803234 +0000 UTC m=+2945.859095500" watchObservedRunningTime="2026-11-02 18:18:49.370069699 +0000 UTC m=+2945.861132859"
Nov 02 18:18:50.332956 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:50.332919    4456 generic.go:356] "Generic (PLEG): container finished" podID="73cfd712-6978-4e5b-a5a2-1641533d46d4" containerID="8d3998bd52036ca16be6d5cd29056cdd91c2e28594d07f9b89ebce19e09bba4c" exitCode=0
Nov 02 18:18:50.333361 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:50.332998    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz" event={"ID":"73cfd712-6978-4e5b-a5a2-1641533d46d4","Type":"ContainerDied","Data":"8d3998bd52036ca16be6d5cd29056cdd91c2e28594d07f9b89ebce19e09bba4c"}
Nov 02 18:18:51.480758 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:51.480723    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"
Nov 02 18:18:51.603300 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:51.603228    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/73cfd712-6978-4e5b-a5a2-1641533d46d4-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73cfd712-6978-4e5b-a5a2-1641533d46d4-maas-api-xks-ca\") pod \"73cfd712-6978-4e5b-a5a2-1641533d46d4\" (UID: \"73cfd712-6978-4e5b-a5a2-1641533d46d4\") "
Nov 02 18:18:51.603435 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:51.603301    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/73cfd712-6978-4e5b-a5a2-1641533d46d4-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73cfd712-6978-4e5b-a5a2-1641533d46d4-maas-api-openshift-ca\") pod \"73cfd712-6978-4e5b-a5a2-1641533d46d4\" (UID: \"73cfd712-6978-4e5b-a5a2-1641533d46d4\") "
Nov 02 18:18:51.603474 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:51.603440    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/73cfd712-6978-4e5b-a5a2-1641533d46d4-kube-api-access-g627f\" (UniqueName: \"kubernetes.io/projected/73cfd712-6978-4e5b-a5a2-1641533d46d4-kube-api-access-g627f\") pod \"73cfd712-6978-4e5b-a5a2-1641533d46d4\" (UID: \"73cfd712-6978-4e5b-a5a2-1641533d46d4\") "
Nov 02 18:18:51.603596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:51.603575    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/73cfd712-6978-4e5b-a5a2-1641533d46d4-maas-api-openshift-ca" pod "73cfd712-6978-4e5b-a5a2-1641533d46d4" (UID: "73cfd712-6978-4e5b-a5a2-1641533d46d4"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:18:51.603768 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:51.603739    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/73cfd712-6978-4e5b-a5a2-1641533d46d4-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:18:51.605504 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:51.605483    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/73cfd712-6978-4e5b-a5a2-1641533d46d4-maas-api-xks-ca" pod "73cfd712-6978-4e5b-a5a2-1641533d46d4" (UID: "73cfd712-6978-4e5b-a5a2-1641533d46d4"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:18:51.605585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:51.605564    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/73cfd712-6978-4e5b-a5a2-1641533d46d4-kube-api-access-g627f" pod "73cfd712-6978-4e5b-a5a2-1641533d46d4" (UID: "73cfd712-6978-4e5b-a5a2-1641533d46d4"). InnerVolumeSpecName "kube-api-access-g627f". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:18:51.705260 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:51.705234    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g627f\" (UniqueName: \"kubernetes.io/projected/73cfd712-6978-4e5b-a5a2-1641533d46d4-kube-api-access-g627f\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:18:51.705260 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:51.705263    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/73cfd712-6978-4e5b-a5a2-1641533d46d4-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:18:52.345900 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:52.345858    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz" event={"ID":"73cfd712-6978-4e5b-a5a2-1641533d46d4","Type":"ContainerDied","Data":"2e649eca9af7f8145f16bf963ecb39151745eb6b90af931657f0c1a8ee171e7c"}
Nov 02 18:18:52.345900 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:52.345886    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"
Nov 02 18:18:52.346096 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:52.345922    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2e649eca9af7f8145f16bf963ecb39151745eb6b90af931657f0c1a8ee171e7c"
Nov 02 18:18:58.324864 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.324808    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"]
Nov 02 18:18:58.325251 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.325240    4456 cpu_manager.go:414] "containerMap: removing container" podUID="73cfd712-6978-4e5b-a5a2-1641533d46d4" containerName="revoke-keys"
Nov 02 18:18:58.325291 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.325251    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="73cfd712-6978-4e5b-a5a2-1641533d46d4" containerName="revoke-keys"
Nov 02 18:18:58.325329 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.325319    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="73cfd712-6978-4e5b-a5a2-1641533d46d4" containerName="revoke-keys"
Nov 02 18:18:58.340095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.340052    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"
Nov 02 18:18:58.348467 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.348432    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"]
Nov 02 18:18:58.359378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.359338    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:18:58.359514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.359339    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:18:58.359514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.359434    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:18:58.372645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.372603    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:18:58.470373 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.470332    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dccc5fbb-b40d-4e99-8195-24cb2440c21a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-vtwgc\" (UID: \"dccc5fbb-b40d-4e99-8195-24cb2440c21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"
Nov 02 18:18:58.470373 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.470375    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dccc5fbb-b40d-4e99-8195-24cb2440c21a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-vtwgc\" (UID: \"dccc5fbb-b40d-4e99-8195-24cb2440c21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"
Nov 02 18:18:58.470623 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.470450    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2tfhc\" (UniqueName: \"kubernetes.io/projected/dccc5fbb-b40d-4e99-8195-24cb2440c21a-kube-api-access-2tfhc\") pod \"maas-api-revoke-sub-d3fba056-vtwgc\" (UID: \"dccc5fbb-b40d-4e99-8195-24cb2440c21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"
Nov 02 18:18:58.571966 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.571926    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dccc5fbb-b40d-4e99-8195-24cb2440c21a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-vtwgc\" (UID: \"dccc5fbb-b40d-4e99-8195-24cb2440c21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"
Nov 02 18:18:58.572153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.571976    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dccc5fbb-b40d-4e99-8195-24cb2440c21a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-vtwgc\" (UID: \"dccc5fbb-b40d-4e99-8195-24cb2440c21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"
Nov 02 18:18:58.572153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.572027    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2tfhc\" (UniqueName: \"kubernetes.io/projected/dccc5fbb-b40d-4e99-8195-24cb2440c21a-kube-api-access-2tfhc\") pod \"maas-api-revoke-sub-d3fba056-vtwgc\" (UID: \"dccc5fbb-b40d-4e99-8195-24cb2440c21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"
Nov 02 18:18:58.572653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.572627    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dccc5fbb-b40d-4e99-8195-24cb2440c21a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d3fba056-vtwgc\" (UID: \"dccc5fbb-b40d-4e99-8195-24cb2440c21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"
Nov 02 18:18:58.574593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.574571    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dccc5fbb-b40d-4e99-8195-24cb2440c21a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d3fba056-vtwgc\" (UID: \"dccc5fbb-b40d-4e99-8195-24cb2440c21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"
Nov 02 18:18:58.596581 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.596524    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2tfhc\" (UniqueName: \"kubernetes.io/projected/dccc5fbb-b40d-4e99-8195-24cb2440c21a-kube-api-access-2tfhc\") pod \"maas-api-revoke-sub-d3fba056-vtwgc\" (UID: \"dccc5fbb-b40d-4e99-8195-24cb2440c21a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"
Nov 02 18:18:58.656087 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.656064    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"
Nov 02 18:18:58.953287 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:58.953256    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"]
Nov 02 18:18:58.954417 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:18:58.954384    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poddccc5fbb_b40d_4e99_8195_24cb2440c21a.slice/crio-59e92a111575b737ec65eff33210b2e4e859e5348de351a67f23ad14fd1882fb WatchSource:0}: Error finding container 59e92a111575b737ec65eff33210b2e4e859e5348de351a67f23ad14fd1882fb: Status 404 returned error can't find the container with id 59e92a111575b737ec65eff33210b2e4e859e5348de351a67f23ad14fd1882fb
Nov 02 18:18:59.250027 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:59.249942    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:18:59.390938 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:59.390909    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc" event={"ID":"dccc5fbb-b40d-4e99-8195-24cb2440c21a","Type":"ContainerStarted","Data":"59e92a111575b737ec65eff33210b2e4e859e5348de351a67f23ad14fd1882fb"}
Nov 02 18:18:59.400185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:18:59.400138    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:00.401142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:00.401043    4456 generic.go:356] "Generic (PLEG): container finished" podID="dccc5fbb-b40d-4e99-8195-24cb2440c21a" containerID="6f00e4f1b990da667fb5172c8ce252417cd8e6f9a40b43ffd2b4693c83afaa3e" exitCode=0
Nov 02 18:19:00.401607 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:00.401247    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc" event={"ID":"dccc5fbb-b40d-4e99-8195-24cb2440c21a","Type":"ContainerDied","Data":"6f00e4f1b990da667fb5172c8ce252417cd8e6f9a40b43ffd2b4693c83afaa3e"}
Nov 02 18:19:01.882093 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:01.882065    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"
Nov 02 18:19:01.980359 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:01.980326    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/dccc5fbb-b40d-4e99-8195-24cb2440c21a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dccc5fbb-b40d-4e99-8195-24cb2440c21a-maas-api-xks-ca\") pod \"dccc5fbb-b40d-4e99-8195-24cb2440c21a\" (UID: \"dccc5fbb-b40d-4e99-8195-24cb2440c21a\") "
Nov 02 18:19:01.988557 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:01.988126    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/dccc5fbb-b40d-4e99-8195-24cb2440c21a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dccc5fbb-b40d-4e99-8195-24cb2440c21a-maas-api-openshift-ca\") pod \"dccc5fbb-b40d-4e99-8195-24cb2440c21a\" (UID: \"dccc5fbb-b40d-4e99-8195-24cb2440c21a\") "
Nov 02 18:19:01.988865 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:01.988511    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/dccc5fbb-b40d-4e99-8195-24cb2440c21a-maas-api-openshift-ca" pod "dccc5fbb-b40d-4e99-8195-24cb2440c21a" (UID: "dccc5fbb-b40d-4e99-8195-24cb2440c21a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:19:01.989344 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:01.989324    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/dccc5fbb-b40d-4e99-8195-24cb2440c21a-kube-api-access-2tfhc\" (UniqueName: \"kubernetes.io/projected/dccc5fbb-b40d-4e99-8195-24cb2440c21a-kube-api-access-2tfhc\") pod \"dccc5fbb-b40d-4e99-8195-24cb2440c21a\" (UID: \"dccc5fbb-b40d-4e99-8195-24cb2440c21a\") "
Nov 02 18:19:02.000838 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:01.999047    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/dccc5fbb-b40d-4e99-8195-24cb2440c21a-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:19:02.028864 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:02.008315    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/dccc5fbb-b40d-4e99-8195-24cb2440c21a-kube-api-access-2tfhc" pod "dccc5fbb-b40d-4e99-8195-24cb2440c21a" (UID: "dccc5fbb-b40d-4e99-8195-24cb2440c21a"). InnerVolumeSpecName "kube-api-access-2tfhc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:19:02.028864 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:02.010271    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/dccc5fbb-b40d-4e99-8195-24cb2440c21a-maas-api-xks-ca" pod "dccc5fbb-b40d-4e99-8195-24cb2440c21a" (UID: "dccc5fbb-b40d-4e99-8195-24cb2440c21a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:19:02.113901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:02.113528    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/dccc5fbb-b40d-4e99-8195-24cb2440c21a-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:19:02.113901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:02.113566    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2tfhc\" (UniqueName: \"kubernetes.io/projected/dccc5fbb-b40d-4e99-8195-24cb2440c21a-kube-api-access-2tfhc\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:19:02.432312 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:02.432228    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"
Nov 02 18:19:02.432312 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:02.432262    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc" event={"ID":"dccc5fbb-b40d-4e99-8195-24cb2440c21a","Type":"ContainerDied","Data":"59e92a111575b737ec65eff33210b2e4e859e5348de351a67f23ad14fd1882fb"}
Nov 02 18:19:02.432312 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:02.432307    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="59e92a111575b737ec65eff33210b2e4e859e5348de351a67f23ad14fd1882fb"
Nov 02 18:19:11.475133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:11.475096    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:13.651183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:13.651153    4456 scope.go:122] "RemoveContainer" containerID="256a789ee8ff0b05eeb0b9206df884875f6c586a375dac7962d8407fdab28a9c"
Nov 02 18:19:13.662873 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:13.662850    4456 scope.go:122] "RemoveContainer" containerID="08f8867e9ea760910a33ddc37130b5a36993599b9b7b82fa40374f018983d586"
Nov 02 18:19:13.671278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:13.671258    4456 scope.go:122] "RemoveContainer" containerID="3e1f290982f9eb51c9746d0938082f539788e09c6c42daea51dd25c8474dddae"
Nov 02 18:19:13.679069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:13.679052    4456 scope.go:122] "RemoveContainer" containerID="401ebf16b44e5436d527807df450685a251b2fed001cdf07320abf6b9bf2eb1b"
Nov 02 18:19:13.686685 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:13.686651    4456 scope.go:122] "RemoveContainer" containerID="44af70e06419b72cee419c8f1a1c535d95e4012f225cd305a78ac8ee937de357"
Nov 02 18:19:13.694286 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:13.694267    4456 scope.go:122] "RemoveContainer" containerID="45c8454f3d685f36d4e8376892c0d02950b001de66596fa7dd338b90f71d8aef"
Nov 02 18:19:13.701949 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:13.701926    4456 scope.go:122] "RemoveContainer" containerID="9908738740a4d442ba29a6ddac0a89a05fc249ad9c39bf216ab06f50cf444344"
Nov 02 18:19:13.709509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:13.709493    4456 scope.go:122] "RemoveContainer" containerID="bab74c3384f8f0dec76df071459ea93b500da899e848d8bef8c31655c7321583"
Nov 02 18:19:13.717372 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:13.717337    4456 scope.go:122] "RemoveContainer" containerID="e8183aab1c9e706d4a6b3dea7ef363ecbf7ba4bdb05426c127e5ed0525bee425"
Nov 02 18:19:16.237094 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:19:16.237051    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:19:20.042568 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.042530    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"]
Nov 02 18:19:20.042998 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.042982    4456 cpu_manager.go:414] "containerMap: removing container" podUID="dccc5fbb-b40d-4e99-8195-24cb2440c21a" containerName="revoke-keys"
Nov 02 18:19:20.043041 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.043001    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="dccc5fbb-b40d-4e99-8195-24cb2440c21a" containerName="revoke-keys"
Nov 02 18:19:20.043082 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.043068    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="dccc5fbb-b40d-4e99-8195-24cb2440c21a" containerName="revoke-keys"
Nov 02 18:19:20.046232 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.046215    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"
Nov 02 18:19:20.063532 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.063102    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:19:20.076687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.076653    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:19:20.076687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.076681    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:19:20.076965 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.076664    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:19:20.080497 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.080469    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"]
Nov 02 18:19:20.111060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.111018    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-v5xqz\" (UID: \"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"
Nov 02 18:19:20.111228 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.111138    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-v5xqz\" (UID: \"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"
Nov 02 18:19:20.111277 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.111221    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lsl4z\" (UniqueName: \"kubernetes.io/projected/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-kube-api-access-lsl4z\") pod \"maas-api-revoke-sub-ae594979-v5xqz\" (UID: \"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"
Nov 02 18:19:20.212182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.212147    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-v5xqz\" (UID: \"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"
Nov 02 18:19:20.212182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.212187    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-v5xqz\" (UID: \"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"
Nov 02 18:19:20.212408 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.212340    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lsl4z\" (UniqueName: \"kubernetes.io/projected/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-kube-api-access-lsl4z\") pod \"maas-api-revoke-sub-ae594979-v5xqz\" (UID: \"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"
Nov 02 18:19:20.212787 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.212766    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-ae594979-v5xqz\" (UID: \"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"
Nov 02 18:19:20.214658 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.214633    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-ae594979-v5xqz\" (UID: \"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"
Nov 02 18:19:20.237442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.237419    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lsl4z\" (UniqueName: \"kubernetes.io/projected/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-kube-api-access-lsl4z\") pod \"maas-api-revoke-sub-ae594979-v5xqz\" (UID: \"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"
Nov 02 18:19:20.360461 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.360369    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"
Nov 02 18:19:20.663789 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:20.663762    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"]
Nov 02 18:19:20.664600 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:19:20.664569    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd07e279e_9e65_43b9_8a3e_ea3927f0bf5d.slice/crio-9d15194fd742da071b7f10cbd6ae2250b5470442c3b03d521250c414ce6622b1 WatchSource:0}: Error finding container 9d15194fd742da071b7f10cbd6ae2250b5470442c3b03d521250c414ce6622b1: Status 404 returned error can't find the container with id 9d15194fd742da071b7f10cbd6ae2250b5470442c3b03d521250c414ce6622b1
Nov 02 18:19:21.560743 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:21.560706    4456 generic.go:356] "Generic (PLEG): container finished" podID="d07e279e-9e65-43b9-8a3e-ea3927f0bf5d" containerID="b5bd2f95da9c4d77655ae7512e2d4e78e3a21a7ddaf00fa2483d959637990940" exitCode=0
Nov 02 18:19:21.561109 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:21.560794    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz" event={"ID":"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d","Type":"ContainerDied","Data":"b5bd2f95da9c4d77655ae7512e2d4e78e3a21a7ddaf00fa2483d959637990940"}
Nov 02 18:19:21.561109 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:21.560874    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz" event={"ID":"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d","Type":"ContainerStarted","Data":"9d15194fd742da071b7f10cbd6ae2250b5470442c3b03d521250c414ce6622b1"}
Nov 02 18:19:22.697280 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:22.697254    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"
Nov 02 18:19:22.737492 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:22.737465    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-maas-api-openshift-ca\") pod \"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d\" (UID: \"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d\") "
Nov 02 18:19:22.737627 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:22.737513    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-kube-api-access-lsl4z\" (UniqueName: \"kubernetes.io/projected/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-kube-api-access-lsl4z\") pod \"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d\" (UID: \"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d\") "
Nov 02 18:19:22.737627 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:22.737553    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-maas-api-xks-ca\") pod \"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d\" (UID: \"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d\") "
Nov 02 18:19:22.737872 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:22.737846    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-maas-api-openshift-ca" pod "d07e279e-9e65-43b9-8a3e-ea3927f0bf5d" (UID: "d07e279e-9e65-43b9-8a3e-ea3927f0bf5d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:19:22.740268 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:22.740243    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-maas-api-xks-ca" pod "d07e279e-9e65-43b9-8a3e-ea3927f0bf5d" (UID: "d07e279e-9e65-43b9-8a3e-ea3927f0bf5d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:19:22.740623 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:22.740600    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-kube-api-access-lsl4z" pod "d07e279e-9e65-43b9-8a3e-ea3927f0bf5d" (UID: "d07e279e-9e65-43b9-8a3e-ea3927f0bf5d"). InnerVolumeSpecName "kube-api-access-lsl4z". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:19:22.838907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:22.838799    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:19:22.838907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:22.838858    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lsl4z\" (UniqueName: \"kubernetes.io/projected/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-kube-api-access-lsl4z\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:19:22.838907 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:22.838868    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:19:23.574934 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:23.574899    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"
Nov 02 18:19:23.574934 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:23.574927    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz" event={"ID":"d07e279e-9e65-43b9-8a3e-ea3927f0bf5d","Type":"ContainerDied","Data":"9d15194fd742da071b7f10cbd6ae2250b5470442c3b03d521250c414ce6622b1"}
Nov 02 18:19:23.575144 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:23.574971    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="9d15194fd742da071b7f10cbd6ae2250b5470442c3b03d521250c414ce6622b1"
Nov 02 18:19:33.039606 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:33.039569    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"]
Nov 02 18:19:33.052879 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:33.052852    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0bebed52-74hfl"]
Nov 02 18:19:33.613066 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:33.613031    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="10825c71-87c7-4b3d-b09a-f17130b1b391" path="/var/lib/kubelet/pods/10825c71-87c7-4b3d-b09a-f17130b1b391/volumes"
Nov 02 18:19:38.908894 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:38.908855    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:41.548124 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.548086    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"]
Nov 02 18:19:41.548645 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.548626    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d07e279e-9e65-43b9-8a3e-ea3927f0bf5d" containerName="revoke-keys"
Nov 02 18:19:41.548692 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.548649    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d07e279e-9e65-43b9-8a3e-ea3927f0bf5d" containerName="revoke-keys"
Nov 02 18:19:41.548726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.548719    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d07e279e-9e65-43b9-8a3e-ea3927f0bf5d" containerName="revoke-keys"
Nov 02 18:19:41.552979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.552964    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"
Nov 02 18:19:41.571752 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.571712    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"]
Nov 02 18:19:41.581642 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.581600    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:19:41.581642 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.581605    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:19:41.581642 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.581633    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:19:41.581921 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.581668    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:19:41.625225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.625193    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zbqlf\" (UID: \"007a9ed2-f45f-4cd9-b033-2e1ce58088ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"
Nov 02 18:19:41.625375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.625257    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mzksc\" (UniqueName: \"kubernetes.io/projected/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-kube-api-access-mzksc\") pod \"maas-api-revoke-sub-a4aef7d7-zbqlf\" (UID: \"007a9ed2-f45f-4cd9-b033-2e1ce58088ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"
Nov 02 18:19:41.625375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.625341    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zbqlf\" (UID: \"007a9ed2-f45f-4cd9-b033-2e1ce58088ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"
Nov 02 18:19:41.726343 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.726302    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zbqlf\" (UID: \"007a9ed2-f45f-4cd9-b033-2e1ce58088ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"
Nov 02 18:19:41.726508 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.726376    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mzksc\" (UniqueName: \"kubernetes.io/projected/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-kube-api-access-mzksc\") pod \"maas-api-revoke-sub-a4aef7d7-zbqlf\" (UID: \"007a9ed2-f45f-4cd9-b033-2e1ce58088ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"
Nov 02 18:19:41.726508 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.726402    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zbqlf\" (UID: \"007a9ed2-f45f-4cd9-b033-2e1ce58088ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"
Nov 02 18:19:41.727152 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.727133    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zbqlf\" (UID: \"007a9ed2-f45f-4cd9-b033-2e1ce58088ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"
Nov 02 18:19:41.728772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.728753    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-maas-api-xks-ca\") pod \"maas-api-revoke-sub-a4aef7d7-zbqlf\" (UID: \"007a9ed2-f45f-4cd9-b033-2e1ce58088ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"
Nov 02 18:19:41.750787 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.750765    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mzksc\" (UniqueName: \"kubernetes.io/projected/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-kube-api-access-mzksc\") pod \"maas-api-revoke-sub-a4aef7d7-zbqlf\" (UID: \"007a9ed2-f45f-4cd9-b033-2e1ce58088ca\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"
Nov 02 18:19:41.865887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:41.865798    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"
Nov 02 18:19:42.164081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:42.164052    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"]
Nov 02 18:19:42.164938 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:19:42.164901    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod007a9ed2_f45f_4cd9_b033_2e1ce58088ca.slice/crio-ca32b3520a182c8f57422d62b1fb7c4849ee552e574bbb1ba4267209c6b25b40 WatchSource:0}: Error finding container ca32b3520a182c8f57422d62b1fb7c4849ee552e574bbb1ba4267209c6b25b40: Status 404 returned error can't find the container with id ca32b3520a182c8f57422d62b1fb7c4849ee552e574bbb1ba4267209c6b25b40
Nov 02 18:19:42.704401 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:42.704366    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf" event={"ID":"007a9ed2-f45f-4cd9-b033-2e1ce58088ca","Type":"ContainerStarted","Data":"718dc6dfa59704bf6ed0ccfd19ff998fb4b685a22684857200a0cac92e794130"}
Nov 02 18:19:42.704794 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:42.704418    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf" event={"ID":"007a9ed2-f45f-4cd9-b033-2e1ce58088ca","Type":"ContainerStarted","Data":"ca32b3520a182c8f57422d62b1fb7c4849ee552e574bbb1ba4267209c6b25b40"}
Nov 02 18:19:42.748680 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:42.748589    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf" podStartSLOduration=1.7485773770000002 podStartE2EDuration="1.748577377s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:19:41 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:19:42.746588003 +0000 UTC m=+2999.237651183" watchObservedRunningTime="2026-11-02 18:19:42.748577377 +0000 UTC m=+2999.239640538"
Nov 02 18:19:43.715625 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:43.715593    4456 generic.go:356] "Generic (PLEG): container finished" podID="007a9ed2-f45f-4cd9-b033-2e1ce58088ca" containerID="718dc6dfa59704bf6ed0ccfd19ff998fb4b685a22684857200a0cac92e794130" exitCode=0
Nov 02 18:19:43.716019 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:43.715679    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf" event={"ID":"007a9ed2-f45f-4cd9-b033-2e1ce58088ca","Type":"ContainerDied","Data":"718dc6dfa59704bf6ed0ccfd19ff998fb4b685a22684857200a0cac92e794130"}
Nov 02 18:19:44.031017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:44.030924    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"]
Nov 02 18:19:44.033547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:44.033524    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-46e4abcd-4dz8j"]
Nov 02 18:19:44.858583 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:44.858560    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"
Nov 02 18:19:44.958614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:44.958574    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-maas-api-xks-ca\") pod \"007a9ed2-f45f-4cd9-b033-2e1ce58088ca\" (UID: \"007a9ed2-f45f-4cd9-b033-2e1ce58088ca\") "
Nov 02 18:19:44.958851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:44.958635    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-kube-api-access-mzksc\" (UniqueName: \"kubernetes.io/projected/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-kube-api-access-mzksc\") pod \"007a9ed2-f45f-4cd9-b033-2e1ce58088ca\" (UID: \"007a9ed2-f45f-4cd9-b033-2e1ce58088ca\") "
Nov 02 18:19:44.958851 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:44.958670    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-maas-api-openshift-ca\") pod \"007a9ed2-f45f-4cd9-b033-2e1ce58088ca\" (UID: \"007a9ed2-f45f-4cd9-b033-2e1ce58088ca\") "
Nov 02 18:19:44.959064 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:44.959040    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-maas-api-openshift-ca" pod "007a9ed2-f45f-4cd9-b033-2e1ce58088ca" (UID: "007a9ed2-f45f-4cd9-b033-2e1ce58088ca"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:19:44.961011 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:44.960981    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-maas-api-xks-ca" pod "007a9ed2-f45f-4cd9-b033-2e1ce58088ca" (UID: "007a9ed2-f45f-4cd9-b033-2e1ce58088ca"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:19:44.961105 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:44.961029    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-kube-api-access-mzksc" pod "007a9ed2-f45f-4cd9-b033-2e1ce58088ca" (UID: "007a9ed2-f45f-4cd9-b033-2e1ce58088ca"). InnerVolumeSpecName "kube-api-access-mzksc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:19:45.059517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:45.059435    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:19:45.059517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:45.059463    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mzksc\" (UniqueName: \"kubernetes.io/projected/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-kube-api-access-mzksc\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:19:45.059517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:45.059473    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/007a9ed2-f45f-4cd9-b033-2e1ce58088ca-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:19:45.613947 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:45.613908    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="32271bff-bd2b-43a8-8da1-cc7ae9b43cd7" path="/var/lib/kubelet/pods/32271bff-bd2b-43a8-8da1-cc7ae9b43cd7/volumes"
Nov 02 18:19:45.728967 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:45.728936    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"
Nov 02 18:19:45.729149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:45.728940    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf" event={"ID":"007a9ed2-f45f-4cd9-b033-2e1ce58088ca","Type":"ContainerDied","Data":"ca32b3520a182c8f57422d62b1fb7c4849ee552e574bbb1ba4267209c6b25b40"}
Nov 02 18:19:45.729149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:45.729054    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="ca32b3520a182c8f57422d62b1fb7c4849ee552e574bbb1ba4267209c6b25b40"
Nov 02 18:19:46.164642 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:19:46.164607    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:19:49.686791 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:49.686759    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:59.828840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:59.820019    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:19:59.944840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:59.944007    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-r5jnj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:19:59.944840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:19:59.944236    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-r5jnj" podUID="cce08d5b-d93c-4c5f-bb03-89c460372a06" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.26:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:20:01.811829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:01.811775    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"]
Nov 02 18:20:01.812303 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:01.812286    4456 cpu_manager.go:414] "containerMap: removing container" podUID="007a9ed2-f45f-4cd9-b033-2e1ce58088ca" containerName="revoke-keys"
Nov 02 18:20:01.812346 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:01.812306    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="007a9ed2-f45f-4cd9-b033-2e1ce58088ca" containerName="revoke-keys"
Nov 02 18:20:01.812379 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:01.812371    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="007a9ed2-f45f-4cd9-b033-2e1ce58088ca" containerName="revoke-keys"
Nov 02 18:20:01.816536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:01.816518    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"
Nov 02 18:20:01.835387 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:01.835355    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"]
Nov 02 18:20:01.846016 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:01.845990    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:20:01.846016 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:01.846014    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:20:01.846227 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:01.846042    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:20:01.847288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:01.847264    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:20:01.958909 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:01.958873    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-g974v\" (UniqueName: \"kubernetes.io/projected/fda269f0-9046-47e3-ae35-c4469ddf41f7-kube-api-access-g974v\") pod \"maas-api-revoke-sub-79805ca9-6fhn9\" (UID: \"fda269f0-9046-47e3-ae35-c4469ddf41f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"
Nov 02 18:20:01.959072 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:01.959005    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda269f0-9046-47e3-ae35-c4469ddf41f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6fhn9\" (UID: \"fda269f0-9046-47e3-ae35-c4469ddf41f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"
Nov 02 18:20:01.959072 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:01.959047    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda269f0-9046-47e3-ae35-c4469ddf41f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6fhn9\" (UID: \"fda269f0-9046-47e3-ae35-c4469ddf41f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"
Nov 02 18:20:02.060263 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:02.060228    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda269f0-9046-47e3-ae35-c4469ddf41f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6fhn9\" (UID: \"fda269f0-9046-47e3-ae35-c4469ddf41f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"
Nov 02 18:20:02.060412 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:02.060285    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda269f0-9046-47e3-ae35-c4469ddf41f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6fhn9\" (UID: \"fda269f0-9046-47e3-ae35-c4469ddf41f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"
Nov 02 18:20:02.060412 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:02.060340    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-g974v\" (UniqueName: \"kubernetes.io/projected/fda269f0-9046-47e3-ae35-c4469ddf41f7-kube-api-access-g974v\") pod \"maas-api-revoke-sub-79805ca9-6fhn9\" (UID: \"fda269f0-9046-47e3-ae35-c4469ddf41f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"
Nov 02 18:20:02.061153 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:02.061083    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda269f0-9046-47e3-ae35-c4469ddf41f7-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-79805ca9-6fhn9\" (UID: \"fda269f0-9046-47e3-ae35-c4469ddf41f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"
Nov 02 18:20:02.062782 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:02.062728    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda269f0-9046-47e3-ae35-c4469ddf41f7-maas-api-xks-ca\") pod \"maas-api-revoke-sub-79805ca9-6fhn9\" (UID: \"fda269f0-9046-47e3-ae35-c4469ddf41f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"
Nov 02 18:20:02.085383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:02.085348    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-g974v\" (UniqueName: \"kubernetes.io/projected/fda269f0-9046-47e3-ae35-c4469ddf41f7-kube-api-access-g974v\") pod \"maas-api-revoke-sub-79805ca9-6fhn9\" (UID: \"fda269f0-9046-47e3-ae35-c4469ddf41f7\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"
Nov 02 18:20:02.130170 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:02.130145    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"
Nov 02 18:20:02.431327 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:02.431303    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"]
Nov 02 18:20:02.432144 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:20:02.432116    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podfda269f0_9046_47e3_ae35_c4469ddf41f7.slice/crio-016249cbbb0d4949249cfe90916086427fbe3c393d94e68e41b4ce6f5f1d0aeb WatchSource:0}: Error finding container 016249cbbb0d4949249cfe90916086427fbe3c393d94e68e41b4ce6f5f1d0aeb: Status 404 returned error can't find the container with id 016249cbbb0d4949249cfe90916086427fbe3c393d94e68e41b4ce6f5f1d0aeb
Nov 02 18:20:02.853871 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:02.853836    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9" event={"ID":"fda269f0-9046-47e3-ae35-c4469ddf41f7","Type":"ContainerStarted","Data":"236134598a43af8d1d5f9fd29245771288fdaf2bfb9c3bec0dd1bab979b56ffb"}
Nov 02 18:20:02.854230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:02.853897    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9" event={"ID":"fda269f0-9046-47e3-ae35-c4469ddf41f7","Type":"ContainerStarted","Data":"016249cbbb0d4949249cfe90916086427fbe3c393d94e68e41b4ce6f5f1d0aeb"}
Nov 02 18:20:02.910722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:02.910676    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9" podStartSLOduration=1.910663498 podStartE2EDuration="1.910663498s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:20:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:20:02.910056281 +0000 UTC m=+3019.401119443" watchObservedRunningTime="2026-11-02 18:20:02.910663498 +0000 UTC m=+3019.401726684"
Nov 02 18:20:03.861638 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:03.861600    4456 generic.go:356] "Generic (PLEG): container finished" podID="fda269f0-9046-47e3-ae35-c4469ddf41f7" containerID="236134598a43af8d1d5f9fd29245771288fdaf2bfb9c3bec0dd1bab979b56ffb" exitCode=0
Nov 02 18:20:03.862024 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:03.861669    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9" event={"ID":"fda269f0-9046-47e3-ae35-c4469ddf41f7","Type":"ContainerDied","Data":"236134598a43af8d1d5f9fd29245771288fdaf2bfb9c3bec0dd1bab979b56ffb"}
Nov 02 18:20:05.003553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:05.003526    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"
Nov 02 18:20:05.191733 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:05.191703    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/fda269f0-9046-47e3-ae35-c4469ddf41f7-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda269f0-9046-47e3-ae35-c4469ddf41f7-maas-api-xks-ca\") pod \"fda269f0-9046-47e3-ae35-c4469ddf41f7\" (UID: \"fda269f0-9046-47e3-ae35-c4469ddf41f7\") "
Nov 02 18:20:05.191944 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:05.191788    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/fda269f0-9046-47e3-ae35-c4469ddf41f7-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda269f0-9046-47e3-ae35-c4469ddf41f7-maas-api-openshift-ca\") pod \"fda269f0-9046-47e3-ae35-c4469ddf41f7\" (UID: \"fda269f0-9046-47e3-ae35-c4469ddf41f7\") "
Nov 02 18:20:05.192021 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:05.191956    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/fda269f0-9046-47e3-ae35-c4469ddf41f7-kube-api-access-g974v\" (UniqueName: \"kubernetes.io/projected/fda269f0-9046-47e3-ae35-c4469ddf41f7-kube-api-access-g974v\") pod \"fda269f0-9046-47e3-ae35-c4469ddf41f7\" (UID: \"fda269f0-9046-47e3-ae35-c4469ddf41f7\") "
Nov 02 18:20:05.192130 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:05.192107    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/fda269f0-9046-47e3-ae35-c4469ddf41f7-maas-api-openshift-ca" pod "fda269f0-9046-47e3-ae35-c4469ddf41f7" (UID: "fda269f0-9046-47e3-ae35-c4469ddf41f7"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:20:05.192309 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:05.192289    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/fda269f0-9046-47e3-ae35-c4469ddf41f7-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:20:05.194042 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:05.194022    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/fda269f0-9046-47e3-ae35-c4469ddf41f7-maas-api-xks-ca" pod "fda269f0-9046-47e3-ae35-c4469ddf41f7" (UID: "fda269f0-9046-47e3-ae35-c4469ddf41f7"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:20:05.194181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:05.194160    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/fda269f0-9046-47e3-ae35-c4469ddf41f7-kube-api-access-g974v" pod "fda269f0-9046-47e3-ae35-c4469ddf41f7" (UID: "fda269f0-9046-47e3-ae35-c4469ddf41f7"). InnerVolumeSpecName "kube-api-access-g974v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:20:05.293704 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:05.293671    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/fda269f0-9046-47e3-ae35-c4469ddf41f7-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:20:05.293704 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:05.293699    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-g974v\" (UniqueName: \"kubernetes.io/projected/fda269f0-9046-47e3-ae35-c4469ddf41f7-kube-api-access-g974v\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:20:05.875422 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:05.875345    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"
Nov 02 18:20:05.875422 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:05.875364    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9" event={"ID":"fda269f0-9046-47e3-ae35-c4469ddf41f7","Type":"ContainerDied","Data":"016249cbbb0d4949249cfe90916086427fbe3c393d94e68e41b4ce6f5f1d0aeb"}
Nov 02 18:20:05.875422 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:05.875409    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="016249cbbb0d4949249cfe90916086427fbe3c393d94e68e41b4ce6f5f1d0aeb"
Nov 02 18:20:13.772754 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:13.772671    4456 scope.go:122] "RemoveContainer" containerID="aabfa3c98cb11dbc7499d020d1614e4c598229153040f81cd5c7880778664361"
Nov 02 18:20:13.783706 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:13.783681    4456 scope.go:122] "RemoveContainer" containerID="cb76a2fc0415f3de7c9aff049e81d8ba17af8974170662e2fee6da448986a4be"
Nov 02 18:20:13.791951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:13.791931    4456 scope.go:122] "RemoveContainer" containerID="8568021d7fe7c61c983b4ad67a269994f91f4881e20e8f9ee6516ab6dd0bf82a"
Nov 02 18:20:13.802395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:13.802377    4456 scope.go:122] "RemoveContainer" containerID="c8afec9f578891af7df845dff827ad6f16edb17d077bf006546ff0080c7005a1"
Nov 02 18:20:13.810878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:13.810858    4456 scope.go:122] "RemoveContainer" containerID="5aba2d1b65f569de765561cf90205073daed5f1814dd4b9383bdc6e67ac6aa57"
Nov 02 18:20:13.818704 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:13.818684    4456 scope.go:122] "RemoveContainer" containerID="8ecf27952d156cd1e1347837f3c19cb86b6ce3e4619982d42429340e53ef7aed"
Nov 02 18:20:13.826512 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:13.826493    4456 scope.go:122] "RemoveContainer" containerID="52616e876db9daa613fa24f0dd3b85b8ef0d8b80816a4fb006e00af3d5f01eb2"
Nov 02 18:20:13.835600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:13.835577    4456 scope.go:122] "RemoveContainer" containerID="b10406aea9ff1dbe6171294ed32b2f6cc04d1bd3e9c15d758a72762280786288"
Nov 02 18:20:13.843358 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:13.843327    4456 scope.go:122] "RemoveContainer" containerID="590887138fef93f2da3a68becf81f54c2a49fd6f83725ff4291142fd190e6540"
Nov 02 18:20:16.177305 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:20:16.177259    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:20:21.893984 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:21.893950    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"]
Nov 02 18:20:21.894575 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:21.894553    4456 cpu_manager.go:414] "containerMap: removing container" podUID="fda269f0-9046-47e3-ae35-c4469ddf41f7" containerName="revoke-keys"
Nov 02 18:20:21.894667 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:21.894578    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="fda269f0-9046-47e3-ae35-c4469ddf41f7" containerName="revoke-keys"
Nov 02 18:20:21.894724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:21.894686    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="fda269f0-9046-47e3-ae35-c4469ddf41f7" containerName="revoke-keys"
Nov 02 18:20:21.898415 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:21.898379    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"
Nov 02 18:20:21.918227 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:21.918197    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"]
Nov 02 18:20:21.927677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:21.927649    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:20:21.927677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:21.927648    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:20:21.927873 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:21.927707    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:20:21.927873 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:21.927734    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:20:21.998224 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:21.998183    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-7twtd\" (UniqueName: \"kubernetes.io/projected/824f80b5-91a8-481e-a1c7-187595cff9fa-kube-api-access-7twtd\") pod \"maas-api-revoke-sub-0b7e8edb-tl5xf\" (UID: \"824f80b5-91a8-481e-a1c7-187595cff9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"
Nov 02 18:20:21.998224 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:21.998228    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824f80b5-91a8-481e-a1c7-187595cff9fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tl5xf\" (UID: \"824f80b5-91a8-481e-a1c7-187595cff9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"
Nov 02 18:20:21.998490 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:21.998419    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824f80b5-91a8-481e-a1c7-187595cff9fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tl5xf\" (UID: \"824f80b5-91a8-481e-a1c7-187595cff9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"
Nov 02 18:20:22.100303 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:22.099928    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824f80b5-91a8-481e-a1c7-187595cff9fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tl5xf\" (UID: \"824f80b5-91a8-481e-a1c7-187595cff9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"
Nov 02 18:20:22.100303 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:22.100024    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-7twtd\" (UniqueName: \"kubernetes.io/projected/824f80b5-91a8-481e-a1c7-187595cff9fa-kube-api-access-7twtd\") pod \"maas-api-revoke-sub-0b7e8edb-tl5xf\" (UID: \"824f80b5-91a8-481e-a1c7-187595cff9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"
Nov 02 18:20:22.100303 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:22.100082    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824f80b5-91a8-481e-a1c7-187595cff9fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tl5xf\" (UID: \"824f80b5-91a8-481e-a1c7-187595cff9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"
Nov 02 18:20:22.103849 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:22.103589    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824f80b5-91a8-481e-a1c7-187595cff9fa-maas-api-xks-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tl5xf\" (UID: \"824f80b5-91a8-481e-a1c7-187595cff9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"
Nov 02 18:20:22.105881 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:22.104674    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824f80b5-91a8-481e-a1c7-187595cff9fa-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-0b7e8edb-tl5xf\" (UID: \"824f80b5-91a8-481e-a1c7-187595cff9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"
Nov 02 18:20:22.127920 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:22.127887    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-7twtd\" (UniqueName: \"kubernetes.io/projected/824f80b5-91a8-481e-a1c7-187595cff9fa-kube-api-access-7twtd\") pod \"maas-api-revoke-sub-0b7e8edb-tl5xf\" (UID: \"824f80b5-91a8-481e-a1c7-187595cff9fa\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"
Nov 02 18:20:22.216270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:22.216240    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"
Nov 02 18:20:22.513137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:22.513106    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"]
Nov 02 18:20:22.513706 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:20:22.513676    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod824f80b5_91a8_481e_a1c7_187595cff9fa.slice/crio-d5484b4ee09152535eb37b725b7f1a05c9915a0c391bf22481e0fd335cc2a914 WatchSource:0}: Error finding container d5484b4ee09152535eb37b725b7f1a05c9915a0c391bf22481e0fd335cc2a914: Status 404 returned error can't find the container with id d5484b4ee09152535eb37b725b7f1a05c9915a0c391bf22481e0fd335cc2a914
Nov 02 18:20:22.996463 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:22.996420    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf" event={"ID":"824f80b5-91a8-481e-a1c7-187595cff9fa","Type":"ContainerStarted","Data":"ed7641405fbeccbad1389c47987a6ca34d7908619995bd2d6acd428fe2992a56"}
Nov 02 18:20:22.997038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:22.996487    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf" event={"ID":"824f80b5-91a8-481e-a1c7-187595cff9fa","Type":"ContainerStarted","Data":"d5484b4ee09152535eb37b725b7f1a05c9915a0c391bf22481e0fd335cc2a914"}
Nov 02 18:20:23.040195 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:23.040095    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf" podStartSLOduration=2.04008067 podStartE2EDuration="2.04008067s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:20:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:20:23.039610539 +0000 UTC m=+3039.530673713" watchObservedRunningTime="2026-11-02 18:20:23.04008067 +0000 UTC m=+3039.531143830"
Nov 02 18:20:24.003271 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:24.003236    4456 generic.go:356] "Generic (PLEG): container finished" podID="824f80b5-91a8-481e-a1c7-187595cff9fa" containerID="ed7641405fbeccbad1389c47987a6ca34d7908619995bd2d6acd428fe2992a56" exitCode=0
Nov 02 18:20:24.003677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:24.003304    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf" event={"ID":"824f80b5-91a8-481e-a1c7-187595cff9fa","Type":"ContainerDied","Data":"ed7641405fbeccbad1389c47987a6ca34d7908619995bd2d6acd428fe2992a56"}
Nov 02 18:20:25.144576 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:25.144555    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"
Nov 02 18:20:25.233139 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:25.233112    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/824f80b5-91a8-481e-a1c7-187595cff9fa-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824f80b5-91a8-481e-a1c7-187595cff9fa-maas-api-openshift-ca\") pod \"824f80b5-91a8-481e-a1c7-187595cff9fa\" (UID: \"824f80b5-91a8-481e-a1c7-187595cff9fa\") "
Nov 02 18:20:25.233300 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:25.233223    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/824f80b5-91a8-481e-a1c7-187595cff9fa-kube-api-access-7twtd\" (UniqueName: \"kubernetes.io/projected/824f80b5-91a8-481e-a1c7-187595cff9fa-kube-api-access-7twtd\") pod \"824f80b5-91a8-481e-a1c7-187595cff9fa\" (UID: \"824f80b5-91a8-481e-a1c7-187595cff9fa\") "
Nov 02 18:20:25.233300 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:25.233279    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/824f80b5-91a8-481e-a1c7-187595cff9fa-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824f80b5-91a8-481e-a1c7-187595cff9fa-maas-api-xks-ca\") pod \"824f80b5-91a8-481e-a1c7-187595cff9fa\" (UID: \"824f80b5-91a8-481e-a1c7-187595cff9fa\") "
Nov 02 18:20:25.233525 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:25.233504    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/824f80b5-91a8-481e-a1c7-187595cff9fa-maas-api-openshift-ca" pod "824f80b5-91a8-481e-a1c7-187595cff9fa" (UID: "824f80b5-91a8-481e-a1c7-187595cff9fa"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:20:25.233617 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:25.233602    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/824f80b5-91a8-481e-a1c7-187595cff9fa-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:20:25.235497 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:25.235471    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/824f80b5-91a8-481e-a1c7-187595cff9fa-kube-api-access-7twtd" pod "824f80b5-91a8-481e-a1c7-187595cff9fa" (UID: "824f80b5-91a8-481e-a1c7-187595cff9fa"). InnerVolumeSpecName "kube-api-access-7twtd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:20:25.235497 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:25.235484    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/824f80b5-91a8-481e-a1c7-187595cff9fa-maas-api-xks-ca" pod "824f80b5-91a8-481e-a1c7-187595cff9fa" (UID: "824f80b5-91a8-481e-a1c7-187595cff9fa"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:20:25.334155 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:25.334092    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/824f80b5-91a8-481e-a1c7-187595cff9fa-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:20:25.334155 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:25.334118    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-7twtd\" (UniqueName: \"kubernetes.io/projected/824f80b5-91a8-481e-a1c7-187595cff9fa-kube-api-access-7twtd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:20:26.015476 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:26.015448    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"
Nov 02 18:20:26.015476 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:26.015461    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf" event={"ID":"824f80b5-91a8-481e-a1c7-187595cff9fa","Type":"ContainerDied","Data":"d5484b4ee09152535eb37b725b7f1a05c9915a0c391bf22481e0fd335cc2a914"}
Nov 02 18:20:26.015720 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:26.015525    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d5484b4ee09152535eb37b725b7f1a05c9915a0c391bf22481e0fd335cc2a914"
Nov 02 18:20:32.782320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:32.782266    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"]
Nov 02 18:20:32.782913 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:32.782887    4456 cpu_manager.go:414] "containerMap: removing container" podUID="824f80b5-91a8-481e-a1c7-187595cff9fa" containerName="revoke-keys"
Nov 02 18:20:32.782913 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:32.782911    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="824f80b5-91a8-481e-a1c7-187595cff9fa" containerName="revoke-keys"
Nov 02 18:20:32.783086 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:32.783028    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="824f80b5-91a8-481e-a1c7-187595cff9fa" containerName="revoke-keys"
Nov 02 18:20:32.786382 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:32.786357    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"
Nov 02 18:20:32.816147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:32.816116    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:20:32.816371 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:32.816166    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:20:32.816514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:32.816196    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:20:32.816722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:32.816707    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:20:32.820552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:32.820514    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"]
Nov 02 18:20:32.904955 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:32.904921    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4kfnf\" (UniqueName: \"kubernetes.io/projected/a3b0fff1-0f6c-469d-9f86-921eb4876151-kube-api-access-4kfnf\") pod \"maas-api-revoke-sub-3d4156f1-8wbc7\" (UID: \"a3b0fff1-0f6c-469d-9f86-921eb4876151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"
Nov 02 18:20:32.905120 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:32.904992    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b0fff1-0f6c-469d-9f86-921eb4876151-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8wbc7\" (UID: \"a3b0fff1-0f6c-469d-9f86-921eb4876151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"
Nov 02 18:20:32.905168 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:32.905110    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b0fff1-0f6c-469d-9f86-921eb4876151-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8wbc7\" (UID: \"a3b0fff1-0f6c-469d-9f86-921eb4876151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"
Nov 02 18:20:33.006070 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:33.006039    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b0fff1-0f6c-469d-9f86-921eb4876151-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8wbc7\" (UID: \"a3b0fff1-0f6c-469d-9f86-921eb4876151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"
Nov 02 18:20:33.006242 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:33.006095    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b0fff1-0f6c-469d-9f86-921eb4876151-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8wbc7\" (UID: \"a3b0fff1-0f6c-469d-9f86-921eb4876151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"
Nov 02 18:20:33.006242 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:33.006147    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4kfnf\" (UniqueName: \"kubernetes.io/projected/a3b0fff1-0f6c-469d-9f86-921eb4876151-kube-api-access-4kfnf\") pod \"maas-api-revoke-sub-3d4156f1-8wbc7\" (UID: \"a3b0fff1-0f6c-469d-9f86-921eb4876151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"
Nov 02 18:20:33.006916 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:33.006882    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b0fff1-0f6c-469d-9f86-921eb4876151-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-8wbc7\" (UID: \"a3b0fff1-0f6c-469d-9f86-921eb4876151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"
Nov 02 18:20:33.008802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:33.008774    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b0fff1-0f6c-469d-9f86-921eb4876151-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-8wbc7\" (UID: \"a3b0fff1-0f6c-469d-9f86-921eb4876151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"
Nov 02 18:20:33.031175 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:33.031145    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4kfnf\" (UniqueName: \"kubernetes.io/projected/a3b0fff1-0f6c-469d-9f86-921eb4876151-kube-api-access-4kfnf\") pod \"maas-api-revoke-sub-3d4156f1-8wbc7\" (UID: \"a3b0fff1-0f6c-469d-9f86-921eb4876151\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"
Nov 02 18:20:33.100678 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:33.100596    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"
Nov 02 18:20:33.443994 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:33.443962    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"]
Nov 02 18:20:33.445123 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:20:33.445098    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda3b0fff1_0f6c_469d_9f86_921eb4876151.slice/crio-02133488bc3160866e7a8eade3a440f1677c6909806485cfcfa5c26024e9405f WatchSource:0}: Error finding container 02133488bc3160866e7a8eade3a440f1677c6909806485cfcfa5c26024e9405f: Status 404 returned error can't find the container with id 02133488bc3160866e7a8eade3a440f1677c6909806485cfcfa5c26024e9405f
Nov 02 18:20:34.059116 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:34.059023    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7" event={"ID":"a3b0fff1-0f6c-469d-9f86-921eb4876151","Type":"ContainerStarted","Data":"ac0ca6e8b0f60e96625dd9ef0f3cbbd1fce77906efac8a6598c12fddfd4095c9"}
Nov 02 18:20:34.059116 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:34.059088    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7" event={"ID":"a3b0fff1-0f6c-469d-9f86-921eb4876151","Type":"ContainerStarted","Data":"02133488bc3160866e7a8eade3a440f1677c6909806485cfcfa5c26024e9405f"}
Nov 02 18:20:34.103213 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:34.103162    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7" podStartSLOduration=2.103149012 podStartE2EDuration="2.103149012s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:20:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:20:34.100730802 +0000 UTC m=+3050.591793982" watchObservedRunningTime="2026-11-02 18:20:34.103149012 +0000 UTC m=+3050.594212197"
Nov 02 18:20:35.066723 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:35.066686    4456 generic.go:356] "Generic (PLEG): container finished" podID="a3b0fff1-0f6c-469d-9f86-921eb4876151" containerID="ac0ca6e8b0f60e96625dd9ef0f3cbbd1fce77906efac8a6598c12fddfd4095c9" exitCode=0
Nov 02 18:20:35.067189 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:35.066781    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7" event={"ID":"a3b0fff1-0f6c-469d-9f86-921eb4876151","Type":"ContainerDied","Data":"ac0ca6e8b0f60e96625dd9ef0f3cbbd1fce77906efac8a6598c12fddfd4095c9"}
Nov 02 18:20:36.221900 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:36.221878    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"
Nov 02 18:20:36.342363 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:36.342272    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a3b0fff1-0f6c-469d-9f86-921eb4876151-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b0fff1-0f6c-469d-9f86-921eb4876151-maas-api-xks-ca\") pod \"a3b0fff1-0f6c-469d-9f86-921eb4876151\" (UID: \"a3b0fff1-0f6c-469d-9f86-921eb4876151\") "
Nov 02 18:20:36.342363 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:36.342355    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a3b0fff1-0f6c-469d-9f86-921eb4876151-kube-api-access-4kfnf\" (UniqueName: \"kubernetes.io/projected/a3b0fff1-0f6c-469d-9f86-921eb4876151-kube-api-access-4kfnf\") pod \"a3b0fff1-0f6c-469d-9f86-921eb4876151\" (UID: \"a3b0fff1-0f6c-469d-9f86-921eb4876151\") "
Nov 02 18:20:36.342594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:36.342422    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a3b0fff1-0f6c-469d-9f86-921eb4876151-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b0fff1-0f6c-469d-9f86-921eb4876151-maas-api-openshift-ca\") pod \"a3b0fff1-0f6c-469d-9f86-921eb4876151\" (UID: \"a3b0fff1-0f6c-469d-9f86-921eb4876151\") "
Nov 02 18:20:36.342846 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:36.342796    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a3b0fff1-0f6c-469d-9f86-921eb4876151-maas-api-openshift-ca" pod "a3b0fff1-0f6c-469d-9f86-921eb4876151" (UID: "a3b0fff1-0f6c-469d-9f86-921eb4876151"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:20:36.344495 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:36.344464    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a3b0fff1-0f6c-469d-9f86-921eb4876151-maas-api-xks-ca" pod "a3b0fff1-0f6c-469d-9f86-921eb4876151" (UID: "a3b0fff1-0f6c-469d-9f86-921eb4876151"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:20:36.344653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:36.344637    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a3b0fff1-0f6c-469d-9f86-921eb4876151-kube-api-access-4kfnf" pod "a3b0fff1-0f6c-469d-9f86-921eb4876151" (UID: "a3b0fff1-0f6c-469d-9f86-921eb4876151"). InnerVolumeSpecName "kube-api-access-4kfnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:20:36.444254 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:36.444222    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a3b0fff1-0f6c-469d-9f86-921eb4876151-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:20:36.444254 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:36.444251    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a3b0fff1-0f6c-469d-9f86-921eb4876151-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:20:36.444254 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:36.444261    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4kfnf\" (UniqueName: \"kubernetes.io/projected/a3b0fff1-0f6c-469d-9f86-921eb4876151-kube-api-access-4kfnf\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:20:37.079279 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:37.079251    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"
Nov 02 18:20:37.079443 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:37.079275    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7" event={"ID":"a3b0fff1-0f6c-469d-9f86-921eb4876151","Type":"ContainerDied","Data":"02133488bc3160866e7a8eade3a440f1677c6909806485cfcfa5c26024e9405f"}
Nov 02 18:20:37.079443 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:37.079320    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="02133488bc3160866e7a8eade3a440f1677c6909806485cfcfa5c26024e9405f"
Nov 02 18:20:39.040325 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:39.040286    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:20:46.172560 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:20:46.172489    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:20:56.250930 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.250900    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"]
Nov 02 18:20:56.264136 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.264109    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-8wbc7"]
Nov 02 18:20:56.302598 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.302562    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"]
Nov 02 18:20:56.303099 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.303078    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a3b0fff1-0f6c-469d-9f86-921eb4876151" containerName="revoke-keys"
Nov 02 18:20:56.303099 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.303099    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a3b0fff1-0f6c-469d-9f86-921eb4876151" containerName="revoke-keys"
Nov 02 18:20:56.303259 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.303206    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a3b0fff1-0f6c-469d-9f86-921eb4876151" containerName="revoke-keys"
Nov 02 18:20:56.306422 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.306403    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"
Nov 02 18:20:56.324031 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.323988    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:20:56.324031 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.324012    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:20:56.324232 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.324060    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:20:56.324232 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.324151    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:20:56.325649 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.325624    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"]
Nov 02 18:20:56.409526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.409494    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b6acda-801f-4590-a380-1d671fd18755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xrtrn\" (UID: \"d2b6acda-801f-4590-a380-1d671fd18755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"
Nov 02 18:20:56.409526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.409528    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b6acda-801f-4590-a380-1d671fd18755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xrtrn\" (UID: \"d2b6acda-801f-4590-a380-1d671fd18755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"
Nov 02 18:20:56.409772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.409548    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z4pph\" (UniqueName: \"kubernetes.io/projected/d2b6acda-801f-4590-a380-1d671fd18755-kube-api-access-z4pph\") pod \"maas-api-revoke-sub-3d4156f1-xrtrn\" (UID: \"d2b6acda-801f-4590-a380-1d671fd18755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"
Nov 02 18:20:56.510976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.510882    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b6acda-801f-4590-a380-1d671fd18755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xrtrn\" (UID: \"d2b6acda-801f-4590-a380-1d671fd18755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"
Nov 02 18:20:56.510976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.510931    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b6acda-801f-4590-a380-1d671fd18755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xrtrn\" (UID: \"d2b6acda-801f-4590-a380-1d671fd18755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"
Nov 02 18:20:56.510976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.510966    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z4pph\" (UniqueName: \"kubernetes.io/projected/d2b6acda-801f-4590-a380-1d671fd18755-kube-api-access-z4pph\") pod \"maas-api-revoke-sub-3d4156f1-xrtrn\" (UID: \"d2b6acda-801f-4590-a380-1d671fd18755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"
Nov 02 18:20:56.511622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.511602    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b6acda-801f-4590-a380-1d671fd18755-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-xrtrn\" (UID: \"d2b6acda-801f-4590-a380-1d671fd18755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"
Nov 02 18:20:56.513394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.513373    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b6acda-801f-4590-a380-1d671fd18755-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-xrtrn\" (UID: \"d2b6acda-801f-4590-a380-1d671fd18755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"
Nov 02 18:20:56.535335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.535310    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z4pph\" (UniqueName: \"kubernetes.io/projected/d2b6acda-801f-4590-a380-1d671fd18755-kube-api-access-z4pph\") pod \"maas-api-revoke-sub-3d4156f1-xrtrn\" (UID: \"d2b6acda-801f-4590-a380-1d671fd18755\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"
Nov 02 18:20:56.619473 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.619434    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"
Nov 02 18:20:56.917134 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:56.917106    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"]
Nov 02 18:20:56.918473 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:20:56.918450    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd2b6acda_801f_4590_a380_1d671fd18755.slice/crio-32f1a6ccf0cbb0724c954e4c0fb0a7a68a47a5199518fd72a1d08d454d3cf7b2 WatchSource:0}: Error finding container 32f1a6ccf0cbb0724c954e4c0fb0a7a68a47a5199518fd72a1d08d454d3cf7b2: Status 404 returned error can't find the container with id 32f1a6ccf0cbb0724c954e4c0fb0a7a68a47a5199518fd72a1d08d454d3cf7b2
Nov 02 18:20:57.205714 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:57.205684    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn" event={"ID":"d2b6acda-801f-4590-a380-1d671fd18755","Type":"ContainerStarted","Data":"0157f8fcb991b9a78b3c20b43afd7d18dd2bd63ff3aa7a038270dd33066baa69"}
Nov 02 18:20:57.206038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:57.205735    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn" event={"ID":"d2b6acda-801f-4590-a380-1d671fd18755","Type":"ContainerStarted","Data":"32f1a6ccf0cbb0724c954e4c0fb0a7a68a47a5199518fd72a1d08d454d3cf7b2"}
Nov 02 18:20:57.249221 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:57.249169    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn" podStartSLOduration=1.249154593 podStartE2EDuration="1.249154593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:20:56 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:20:57.247167648 +0000 UTC m=+3073.738230809" watchObservedRunningTime="2026-11-02 18:20:57.249154593 +0000 UTC m=+3073.740217776"
Nov 02 18:20:57.612752 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:57.612674    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a3b0fff1-0f6c-469d-9f86-921eb4876151" path="/var/lib/kubelet/pods/a3b0fff1-0f6c-469d-9f86-921eb4876151/volumes"
Nov 02 18:20:58.220410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:58.220372    4456 generic.go:356] "Generic (PLEG): container finished" podID="d2b6acda-801f-4590-a380-1d671fd18755" containerID="0157f8fcb991b9a78b3c20b43afd7d18dd2bd63ff3aa7a038270dd33066baa69" exitCode=0
Nov 02 18:20:58.220589 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:58.220426    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn" event={"ID":"d2b6acda-801f-4590-a380-1d671fd18755","Type":"ContainerDied","Data":"0157f8fcb991b9a78b3c20b43afd7d18dd2bd63ff3aa7a038270dd33066baa69"}
Nov 02 18:20:59.034191 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:59.034160    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"]
Nov 02 18:20:59.047319 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:59.047288    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-f4b4dd23-6rhns"]
Nov 02 18:20:59.360144 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:59.360121    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"
Nov 02 18:20:59.436933 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:59.436900    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d2b6acda-801f-4590-a380-1d671fd18755-kube-api-access-z4pph\" (UniqueName: \"kubernetes.io/projected/d2b6acda-801f-4590-a380-1d671fd18755-kube-api-access-z4pph\") pod \"d2b6acda-801f-4590-a380-1d671fd18755\" (UID: \"d2b6acda-801f-4590-a380-1d671fd18755\") "
Nov 02 18:20:59.437086 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:59.437008    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d2b6acda-801f-4590-a380-1d671fd18755-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b6acda-801f-4590-a380-1d671fd18755-maas-api-openshift-ca\") pod \"d2b6acda-801f-4590-a380-1d671fd18755\" (UID: \"d2b6acda-801f-4590-a380-1d671fd18755\") "
Nov 02 18:20:59.437127 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:59.437089    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d2b6acda-801f-4590-a380-1d671fd18755-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b6acda-801f-4590-a380-1d671fd18755-maas-api-xks-ca\") pod \"d2b6acda-801f-4590-a380-1d671fd18755\" (UID: \"d2b6acda-801f-4590-a380-1d671fd18755\") "
Nov 02 18:20:59.437438 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:59.437411    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d2b6acda-801f-4590-a380-1d671fd18755-maas-api-openshift-ca" pod "d2b6acda-801f-4590-a380-1d671fd18755" (UID: "d2b6acda-801f-4590-a380-1d671fd18755"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:20:59.439262 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:59.439240    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d2b6acda-801f-4590-a380-1d671fd18755-kube-api-access-z4pph" pod "d2b6acda-801f-4590-a380-1d671fd18755" (UID: "d2b6acda-801f-4590-a380-1d671fd18755"). InnerVolumeSpecName "kube-api-access-z4pph". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:20:59.439366 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:59.439335    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d2b6acda-801f-4590-a380-1d671fd18755-maas-api-xks-ca" pod "d2b6acda-801f-4590-a380-1d671fd18755" (UID: "d2b6acda-801f-4590-a380-1d671fd18755"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:20:59.538926 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:59.538889    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z4pph\" (UniqueName: \"kubernetes.io/projected/d2b6acda-801f-4590-a380-1d671fd18755-kube-api-access-z4pph\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:20:59.538926 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:59.538919    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d2b6acda-801f-4590-a380-1d671fd18755-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:20:59.538926 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:59.538929    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d2b6acda-801f-4590-a380-1d671fd18755-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:20:59.612543 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:20:59.612467    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="573c6b8d-87c3-486a-8a63-1b6b409103e7" path="/var/lib/kubelet/pods/573c6b8d-87c3-486a-8a63-1b6b409103e7/volumes"
Nov 02 18:21:00.232328 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:00.232295    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"
Nov 02 18:21:00.232706 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:00.232304    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn" event={"ID":"d2b6acda-801f-4590-a380-1d671fd18755","Type":"ContainerDied","Data":"32f1a6ccf0cbb0724c954e4c0fb0a7a68a47a5199518fd72a1d08d454d3cf7b2"}
Nov 02 18:21:00.232706 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:00.232424    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="32f1a6ccf0cbb0724c954e4c0fb0a7a68a47a5199518fd72a1d08d454d3cf7b2"
Nov 02 18:21:11.302584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:11.302555    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-driver/0.log"
Nov 02 18:21:11.356785 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:11.356756    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-driver/0.log"
Nov 02 18:21:13.904466 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:13.904429    4456 scope.go:122] "RemoveContainer" containerID="7c8d5209e30f79dfe754b4d00ec10a49f5dd444755c89521e87700bbcc6163ca"
Nov 02 18:21:13.913488 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:13.913464    4456 scope.go:122] "RemoveContainer" containerID="d1a453c9869097ca10c33155469b8043bbc8a267124a052cd5724b3e0dcee129"
Nov 02 18:21:13.921417 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:13.921399    4456 scope.go:122] "RemoveContainer" containerID="d6b3fc0f71a003cf4eb46571b216bb2b835f695c67966d97a48bb40e5c80ba76"
Nov 02 18:21:16.165154 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:21:16.165107    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:21:21.943233 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:21.943199    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"]
Nov 02 18:21:21.943753 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:21.943611    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d2b6acda-801f-4590-a380-1d671fd18755" containerName="revoke-keys"
Nov 02 18:21:21.943753 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:21.943622    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d2b6acda-801f-4590-a380-1d671fd18755" containerName="revoke-keys"
Nov 02 18:21:21.943753 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:21.943693    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d2b6acda-801f-4590-a380-1d671fd18755" containerName="revoke-keys"
Nov 02 18:21:21.947062 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:21.947046    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"
Nov 02 18:21:21.963908 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:21.963497    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:21:21.976838 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:21.976797    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:21:21.977045 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:21.976808    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:21:21.977131 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:21.976922    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:21:21.981154 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:21.981126    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"]
Nov 02 18:21:21.992852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:21.992800    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d889d54c-fb96-4cf0-8d44-17485904481d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qqpfs\" (UID: \"d889d54c-fb96-4cf0-8d44-17485904481d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"
Nov 02 18:21:21.992991 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:21.992960    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq52b\" (UniqueName: \"kubernetes.io/projected/d889d54c-fb96-4cf0-8d44-17485904481d-kube-api-access-jq52b\") pod \"maas-api-revoke-sub-e2fc9c5b-qqpfs\" (UID: \"d889d54c-fb96-4cf0-8d44-17485904481d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"
Nov 02 18:21:21.993095 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:21.993075    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d889d54c-fb96-4cf0-8d44-17485904481d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qqpfs\" (UID: \"d889d54c-fb96-4cf0-8d44-17485904481d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"
Nov 02 18:21:22.094015 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:22.093982    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d889d54c-fb96-4cf0-8d44-17485904481d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qqpfs\" (UID: \"d889d54c-fb96-4cf0-8d44-17485904481d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"
Nov 02 18:21:22.094176 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:22.094034    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq52b\" (UniqueName: \"kubernetes.io/projected/d889d54c-fb96-4cf0-8d44-17485904481d-kube-api-access-jq52b\") pod \"maas-api-revoke-sub-e2fc9c5b-qqpfs\" (UID: \"d889d54c-fb96-4cf0-8d44-17485904481d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"
Nov 02 18:21:22.094176 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:22.094065    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d889d54c-fb96-4cf0-8d44-17485904481d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qqpfs\" (UID: \"d889d54c-fb96-4cf0-8d44-17485904481d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"
Nov 02 18:21:22.094668 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:22.094646    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d889d54c-fb96-4cf0-8d44-17485904481d-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qqpfs\" (UID: \"d889d54c-fb96-4cf0-8d44-17485904481d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"
Nov 02 18:21:22.096465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:22.096446    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d889d54c-fb96-4cf0-8d44-17485904481d-maas-api-xks-ca\") pod \"maas-api-revoke-sub-e2fc9c5b-qqpfs\" (UID: \"d889d54c-fb96-4cf0-8d44-17485904481d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"
Nov 02 18:21:22.118725 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:22.118698    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq52b\" (UniqueName: \"kubernetes.io/projected/d889d54c-fb96-4cf0-8d44-17485904481d-kube-api-access-jq52b\") pod \"maas-api-revoke-sub-e2fc9c5b-qqpfs\" (UID: \"d889d54c-fb96-4cf0-8d44-17485904481d\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"
Nov 02 18:21:22.261047 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:22.260974    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"
Nov 02 18:21:22.558526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:22.558497    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"]
Nov 02 18:21:22.558930 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:21:22.558909    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd889d54c_fb96_4cf0_8d44_17485904481d.slice/crio-53539228656cff76940138078a230f1f793a2de67ea91dabc8f7d727e5c90ff9 WatchSource:0}: Error finding container 53539228656cff76940138078a230f1f793a2de67ea91dabc8f7d727e5c90ff9: Status 404 returned error can't find the container with id 53539228656cff76940138078a230f1f793a2de67ea91dabc8f7d727e5c90ff9
Nov 02 18:21:23.400049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:23.400012    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs" event={"ID":"d889d54c-fb96-4cf0-8d44-17485904481d","Type":"ContainerStarted","Data":"9c62a377bcd5dd7985b529101a480a904b85752a77da4dd783c7c25f86f11bb1"}
Nov 02 18:21:23.400431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:23.400073    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs" event={"ID":"d889d54c-fb96-4cf0-8d44-17485904481d","Type":"ContainerStarted","Data":"53539228656cff76940138078a230f1f793a2de67ea91dabc8f7d727e5c90ff9"}
Nov 02 18:21:23.444290 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:23.444236    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs" podStartSLOduration=2.444222338 podStartE2EDuration="2.444222338s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:21 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:23.442226821 +0000 UTC m=+3099.933289987" watchObservedRunningTime="2026-11-02 18:21:23.444222338 +0000 UTC m=+3099.935285498"
Nov 02 18:21:24.407604 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:24.407570    4456 generic.go:356] "Generic (PLEG): container finished" podID="d889d54c-fb96-4cf0-8d44-17485904481d" containerID="9c62a377bcd5dd7985b529101a480a904b85752a77da4dd783c7c25f86f11bb1" exitCode=0
Nov 02 18:21:24.408089 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:24.407658    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs" event={"ID":"d889d54c-fb96-4cf0-8d44-17485904481d","Type":"ContainerDied","Data":"9c62a377bcd5dd7985b529101a480a904b85752a77da4dd783c7c25f86f11bb1"}
Nov 02 18:21:25.552269 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:25.552242    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"
Nov 02 18:21:25.624399 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:25.624370    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d889d54c-fb96-4cf0-8d44-17485904481d-kube-api-access-jq52b\" (UniqueName: \"kubernetes.io/projected/d889d54c-fb96-4cf0-8d44-17485904481d-kube-api-access-jq52b\") pod \"d889d54c-fb96-4cf0-8d44-17485904481d\" (UID: \"d889d54c-fb96-4cf0-8d44-17485904481d\") "
Nov 02 18:21:25.624584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:25.624415    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d889d54c-fb96-4cf0-8d44-17485904481d-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d889d54c-fb96-4cf0-8d44-17485904481d-maas-api-openshift-ca\") pod \"d889d54c-fb96-4cf0-8d44-17485904481d\" (UID: \"d889d54c-fb96-4cf0-8d44-17485904481d\") "
Nov 02 18:21:25.624584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:25.624499    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/d889d54c-fb96-4cf0-8d44-17485904481d-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d889d54c-fb96-4cf0-8d44-17485904481d-maas-api-xks-ca\") pod \"d889d54c-fb96-4cf0-8d44-17485904481d\" (UID: \"d889d54c-fb96-4cf0-8d44-17485904481d\") "
Nov 02 18:21:25.624800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:25.624776    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d889d54c-fb96-4cf0-8d44-17485904481d-maas-api-openshift-ca" pod "d889d54c-fb96-4cf0-8d44-17485904481d" (UID: "d889d54c-fb96-4cf0-8d44-17485904481d"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:25.626706 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:25.626673    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/d889d54c-fb96-4cf0-8d44-17485904481d-maas-api-xks-ca" pod "d889d54c-fb96-4cf0-8d44-17485904481d" (UID: "d889d54c-fb96-4cf0-8d44-17485904481d"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:21:25.626837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:25.626742    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d889d54c-fb96-4cf0-8d44-17485904481d-kube-api-access-jq52b" pod "d889d54c-fb96-4cf0-8d44-17485904481d" (UID: "d889d54c-fb96-4cf0-8d44-17485904481d"). InnerVolumeSpecName "kube-api-access-jq52b". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:25.726041 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:25.726013    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/d889d54c-fb96-4cf0-8d44-17485904481d-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:21:25.726041 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:25.726039    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/d889d54c-fb96-4cf0-8d44-17485904481d-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:21:25.726041 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:25.726049    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq52b\" (UniqueName: \"kubernetes.io/projected/d889d54c-fb96-4cf0-8d44-17485904481d-kube-api-access-jq52b\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:21:26.420360 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:26.420323    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"
Nov 02 18:21:26.420527 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:26.420355    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs" event={"ID":"d889d54c-fb96-4cf0-8d44-17485904481d","Type":"ContainerDied","Data":"53539228656cff76940138078a230f1f793a2de67ea91dabc8f7d727e5c90ff9"}
Nov 02 18:21:26.420527 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:26.420413    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="53539228656cff76940138078a230f1f793a2de67ea91dabc8f7d727e5c90ff9"
Nov 02 18:21:32.305703 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.305668    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"]
Nov 02 18:21:32.306276 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.306260    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d889d54c-fb96-4cf0-8d44-17485904481d" containerName="revoke-keys"
Nov 02 18:21:32.306321 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.306279    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d889d54c-fb96-4cf0-8d44-17485904481d" containerName="revoke-keys"
Nov 02 18:21:32.306417 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.306406    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d889d54c-fb96-4cf0-8d44-17485904481d" containerName="revoke-keys"
Nov 02 18:21:32.310125 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.310105    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"
Nov 02 18:21:32.327869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.327842    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"]
Nov 02 18:21:32.337976 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.337933    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:21:32.338107 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.338024    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:21:32.339275 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.339257    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:21:32.339364 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.339325    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:21:32.387553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.387517    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11dc527f-427b-4662-a56b-35ee92097ce2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xg6sn\" (UID: \"11dc527f-427b-4662-a56b-35ee92097ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"
Nov 02 18:21:32.387739 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.387593    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-z5dd7\" (UniqueName: \"kubernetes.io/projected/11dc527f-427b-4662-a56b-35ee92097ce2-kube-api-access-z5dd7\") pod \"maas-api-revoke-sub-5a29efb3-xg6sn\" (UID: \"11dc527f-427b-4662-a56b-35ee92097ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"
Nov 02 18:21:32.387739 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.387687    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11dc527f-427b-4662-a56b-35ee92097ce2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xg6sn\" (UID: \"11dc527f-427b-4662-a56b-35ee92097ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"
Nov 02 18:21:32.488626 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.488587    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-z5dd7\" (UniqueName: \"kubernetes.io/projected/11dc527f-427b-4662-a56b-35ee92097ce2-kube-api-access-z5dd7\") pod \"maas-api-revoke-sub-5a29efb3-xg6sn\" (UID: \"11dc527f-427b-4662-a56b-35ee92097ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"
Nov 02 18:21:32.488795 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.488639    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11dc527f-427b-4662-a56b-35ee92097ce2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xg6sn\" (UID: \"11dc527f-427b-4662-a56b-35ee92097ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"
Nov 02 18:21:32.488795 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.488742    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11dc527f-427b-4662-a56b-35ee92097ce2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xg6sn\" (UID: \"11dc527f-427b-4662-a56b-35ee92097ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"
Nov 02 18:21:32.489391 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.489359    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11dc527f-427b-4662-a56b-35ee92097ce2-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-5a29efb3-xg6sn\" (UID: \"11dc527f-427b-4662-a56b-35ee92097ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"
Nov 02 18:21:32.491614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.491590    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11dc527f-427b-4662-a56b-35ee92097ce2-maas-api-xks-ca\") pod \"maas-api-revoke-sub-5a29efb3-xg6sn\" (UID: \"11dc527f-427b-4662-a56b-35ee92097ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"
Nov 02 18:21:32.513393 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.513363    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-z5dd7\" (UniqueName: \"kubernetes.io/projected/11dc527f-427b-4662-a56b-35ee92097ce2-kube-api-access-z5dd7\") pod \"maas-api-revoke-sub-5a29efb3-xg6sn\" (UID: \"11dc527f-427b-4662-a56b-35ee92097ce2\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"
Nov 02 18:21:32.624108 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.624034    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"
Nov 02 18:21:32.923375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:32.923350    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"]
Nov 02 18:21:32.924022 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:21:32.923996    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod11dc527f_427b_4662_a56b_35ee92097ce2.slice/crio-e5b50abd06944f509f838f048a5e1f95d1919bacb770a8d7c9d0987a5bb47e11 WatchSource:0}: Error finding container e5b50abd06944f509f838f048a5e1f95d1919bacb770a8d7c9d0987a5bb47e11: Status 404 returned error can't find the container with id e5b50abd06944f509f838f048a5e1f95d1919bacb770a8d7c9d0987a5bb47e11
Nov 02 18:21:33.462894 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:33.462793    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn" event={"ID":"11dc527f-427b-4662-a56b-35ee92097ce2","Type":"ContainerStarted","Data":"a1c7abedd163731d1dea28fe7bb0a243d7c58b3ab5830334205e61dc72b66a1d"}
Nov 02 18:21:33.462894 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:33.462858    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn" event={"ID":"11dc527f-427b-4662-a56b-35ee92097ce2","Type":"ContainerStarted","Data":"e5b50abd06944f509f838f048a5e1f95d1919bacb770a8d7c9d0987a5bb47e11"}
Nov 02 18:21:33.506598 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:33.506549    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn" podStartSLOduration=1.50653435 podStartE2EDuration="1.50653435s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:32 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:33.504760604 +0000 UTC m=+3109.995823778" watchObservedRunningTime="2026-11-02 18:21:33.50653435 +0000 UTC m=+3109.997597526"
Nov 02 18:21:34.473003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:34.472961    4456 generic.go:356] "Generic (PLEG): container finished" podID="11dc527f-427b-4662-a56b-35ee92097ce2" containerID="a1c7abedd163731d1dea28fe7bb0a243d7c58b3ab5830334205e61dc72b66a1d" exitCode=0
Nov 02 18:21:34.473477 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:34.473031    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn" event={"ID":"11dc527f-427b-4662-a56b-35ee92097ce2","Type":"ContainerDied","Data":"a1c7abedd163731d1dea28fe7bb0a243d7c58b3ab5830334205e61dc72b66a1d"}
Nov 02 18:21:35.901016 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:35.900983    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"
Nov 02 18:21:35.969288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:35.969250    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/11dc527f-427b-4662-a56b-35ee92097ce2-kube-api-access-z5dd7\" (UniqueName: \"kubernetes.io/projected/11dc527f-427b-4662-a56b-35ee92097ce2-kube-api-access-z5dd7\") pod \"11dc527f-427b-4662-a56b-35ee92097ce2\" (UID: \"11dc527f-427b-4662-a56b-35ee92097ce2\") "
Nov 02 18:21:35.969512 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:35.969377    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/11dc527f-427b-4662-a56b-35ee92097ce2-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11dc527f-427b-4662-a56b-35ee92097ce2-maas-api-openshift-ca\") pod \"11dc527f-427b-4662-a56b-35ee92097ce2\" (UID: \"11dc527f-427b-4662-a56b-35ee92097ce2\") "
Nov 02 18:21:35.969512 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:35.969439    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/11dc527f-427b-4662-a56b-35ee92097ce2-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11dc527f-427b-4662-a56b-35ee92097ce2-maas-api-xks-ca\") pod \"11dc527f-427b-4662-a56b-35ee92097ce2\" (UID: \"11dc527f-427b-4662-a56b-35ee92097ce2\") "
Nov 02 18:21:35.971074 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:35.971046    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/11dc527f-427b-4662-a56b-35ee92097ce2-maas-api-openshift-ca" pod "11dc527f-427b-4662-a56b-35ee92097ce2" (UID: "11dc527f-427b-4662-a56b-35ee92097ce2"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:35.972126 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:35.972106    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/11dc527f-427b-4662-a56b-35ee92097ce2-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:21:35.978875 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:35.978848    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/11dc527f-427b-4662-a56b-35ee92097ce2-kube-api-access-z5dd7" pod "11dc527f-427b-4662-a56b-35ee92097ce2" (UID: "11dc527f-427b-4662-a56b-35ee92097ce2"). InnerVolumeSpecName "kube-api-access-z5dd7". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:35.979127 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:35.979103    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/11dc527f-427b-4662-a56b-35ee92097ce2-maas-api-xks-ca" pod "11dc527f-427b-4662-a56b-35ee92097ce2" (UID: "11dc527f-427b-4662-a56b-35ee92097ce2"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:21:36.075637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:36.075558    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-z5dd7\" (UniqueName: \"kubernetes.io/projected/11dc527f-427b-4662-a56b-35ee92097ce2-kube-api-access-z5dd7\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:21:36.075637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:36.075594    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/11dc527f-427b-4662-a56b-35ee92097ce2-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:21:36.486624 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:36.486520    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn" event={"ID":"11dc527f-427b-4662-a56b-35ee92097ce2","Type":"ContainerDied","Data":"e5b50abd06944f509f838f048a5e1f95d1919bacb770a8d7c9d0987a5bb47e11"}
Nov 02 18:21:36.486624 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:36.486595    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="e5b50abd06944f509f838f048a5e1f95d1919bacb770a8d7c9d0987a5bb47e11"
Nov 02 18:21:36.486624 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:36.486593    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"
Nov 02 18:21:39.179658 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:39.179623    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:21:46.165138 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:21:46.165055    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:21:51.494130 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.494093    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"]
Nov 02 18:21:51.507373 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.507346    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-xrtrn"]
Nov 02 18:21:51.546700 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.546670    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"]
Nov 02 18:21:51.547105 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.547091    4456 cpu_manager.go:414] "containerMap: removing container" podUID="11dc527f-427b-4662-a56b-35ee92097ce2" containerName="revoke-keys"
Nov 02 18:21:51.547166 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.547107    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="11dc527f-427b-4662-a56b-35ee92097ce2" containerName="revoke-keys"
Nov 02 18:21:51.547210 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.547198    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="11dc527f-427b-4662-a56b-35ee92097ce2" containerName="revoke-keys"
Nov 02 18:21:51.550365 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.550350    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"
Nov 02 18:21:51.567605 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.567581    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:21:51.567766 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.567619    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:21:51.567935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.567673    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:21:51.567935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.567725    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:21:51.572141 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.572114    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"]
Nov 02 18:21:51.612683 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.612645    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d2b6acda-801f-4590-a380-1d671fd18755" path="/var/lib/kubelet/pods/d2b6acda-801f-4590-a380-1d671fd18755/volumes"
Nov 02 18:21:51.628046 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.628020    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25250034-7a4f-4fff-9efe-543efcd37868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kthn\" (UID: \"25250034-7a4f-4fff-9efe-543efcd37868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"
Nov 02 18:21:51.628152 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.628058    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25250034-7a4f-4fff-9efe-543efcd37868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kthn\" (UID: \"25250034-7a4f-4fff-9efe-543efcd37868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"
Nov 02 18:21:51.628152 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.628087    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-j5r4s\" (UniqueName: \"kubernetes.io/projected/25250034-7a4f-4fff-9efe-543efcd37868-kube-api-access-j5r4s\") pod \"maas-api-revoke-sub-3d4156f1-9kthn\" (UID: \"25250034-7a4f-4fff-9efe-543efcd37868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"
Nov 02 18:21:51.729525 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.729491    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25250034-7a4f-4fff-9efe-543efcd37868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kthn\" (UID: \"25250034-7a4f-4fff-9efe-543efcd37868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"
Nov 02 18:21:51.729708 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.729551    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25250034-7a4f-4fff-9efe-543efcd37868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kthn\" (UID: \"25250034-7a4f-4fff-9efe-543efcd37868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"
Nov 02 18:21:51.729708 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.729594    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-j5r4s\" (UniqueName: \"kubernetes.io/projected/25250034-7a4f-4fff-9efe-543efcd37868-kube-api-access-j5r4s\") pod \"maas-api-revoke-sub-3d4156f1-9kthn\" (UID: \"25250034-7a4f-4fff-9efe-543efcd37868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"
Nov 02 18:21:51.730270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.730249    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25250034-7a4f-4fff-9efe-543efcd37868-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kthn\" (UID: \"25250034-7a4f-4fff-9efe-543efcd37868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"
Nov 02 18:21:51.732062 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.732042    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25250034-7a4f-4fff-9efe-543efcd37868-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-9kthn\" (UID: \"25250034-7a4f-4fff-9efe-543efcd37868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"
Nov 02 18:21:51.754191 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.754135    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-j5r4s\" (UniqueName: \"kubernetes.io/projected/25250034-7a4f-4fff-9efe-543efcd37868-kube-api-access-j5r4s\") pod \"maas-api-revoke-sub-3d4156f1-9kthn\" (UID: \"25250034-7a4f-4fff-9efe-543efcd37868\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"
Nov 02 18:21:51.864122 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:51.864093    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"
Nov 02 18:21:52.162270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:52.162236    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"]
Nov 02 18:21:52.162456 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:21:52.162436    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod25250034_7a4f_4fff_9efe_543efcd37868.slice/crio-d964146422313a4ba6f4f57331b55dacec6e17b11eb1f0611922ddb63d34c704 WatchSource:0}: Error finding container d964146422313a4ba6f4f57331b55dacec6e17b11eb1f0611922ddb63d34c704: Status 404 returned error can't find the container with id d964146422313a4ba6f4f57331b55dacec6e17b11eb1f0611922ddb63d34c704
Nov 02 18:21:52.575561 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:52.575525    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn" event={"ID":"25250034-7a4f-4fff-9efe-543efcd37868","Type":"ContainerStarted","Data":"f5dbb073c90644db2950511066eaf61dc4bc1b3259c75f880fc2ff6bac360a33"}
Nov 02 18:21:52.575960 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:52.575587    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn" event={"ID":"25250034-7a4f-4fff-9efe-543efcd37868","Type":"ContainerStarted","Data":"d964146422313a4ba6f4f57331b55dacec6e17b11eb1f0611922ddb63d34c704"}
Nov 02 18:21:52.619395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:52.619350    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn" podStartSLOduration=1.619337637 podStartE2EDuration="1.619337637s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:21:51 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:21:52.618348411 +0000 UTC m=+3129.109411570" watchObservedRunningTime="2026-11-02 18:21:52.619337637 +0000 UTC m=+3129.110400797"
Nov 02 18:21:53.582346 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:53.582300    4456 generic.go:356] "Generic (PLEG): container finished" podID="25250034-7a4f-4fff-9efe-543efcd37868" containerID="f5dbb073c90644db2950511066eaf61dc4bc1b3259c75f880fc2ff6bac360a33" exitCode=0
Nov 02 18:21:53.582346 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:53.582340    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn" event={"ID":"25250034-7a4f-4fff-9efe-543efcd37868","Type":"ContainerDied","Data":"f5dbb073c90644db2950511066eaf61dc4bc1b3259c75f880fc2ff6bac360a33"}
Nov 02 18:21:54.729335 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:54.729309    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"
Nov 02 18:21:54.858661 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:54.858578    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/25250034-7a4f-4fff-9efe-543efcd37868-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25250034-7a4f-4fff-9efe-543efcd37868-maas-api-openshift-ca\") pod \"25250034-7a4f-4fff-9efe-543efcd37868\" (UID: \"25250034-7a4f-4fff-9efe-543efcd37868\") "
Nov 02 18:21:54.858837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:54.858721    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/25250034-7a4f-4fff-9efe-543efcd37868-kube-api-access-j5r4s\" (UniqueName: \"kubernetes.io/projected/25250034-7a4f-4fff-9efe-543efcd37868-kube-api-access-j5r4s\") pod \"25250034-7a4f-4fff-9efe-543efcd37868\" (UID: \"25250034-7a4f-4fff-9efe-543efcd37868\") "
Nov 02 18:21:54.858837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:54.858756    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/25250034-7a4f-4fff-9efe-543efcd37868-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25250034-7a4f-4fff-9efe-543efcd37868-maas-api-xks-ca\") pod \"25250034-7a4f-4fff-9efe-543efcd37868\" (UID: \"25250034-7a4f-4fff-9efe-543efcd37868\") "
Nov 02 18:21:54.858996 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:54.858973    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/25250034-7a4f-4fff-9efe-543efcd37868-maas-api-openshift-ca" pod "25250034-7a4f-4fff-9efe-543efcd37868" (UID: "25250034-7a4f-4fff-9efe-543efcd37868"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:21:54.859166 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:54.859134    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/25250034-7a4f-4fff-9efe-543efcd37868-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:21:54.861058 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:54.861033    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/25250034-7a4f-4fff-9efe-543efcd37868-kube-api-access-j5r4s" pod "25250034-7a4f-4fff-9efe-543efcd37868" (UID: "25250034-7a4f-4fff-9efe-543efcd37868"). InnerVolumeSpecName "kube-api-access-j5r4s". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:21:54.861128 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:54.861059    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/25250034-7a4f-4fff-9efe-543efcd37868-maas-api-xks-ca" pod "25250034-7a4f-4fff-9efe-543efcd37868" (UID: "25250034-7a4f-4fff-9efe-543efcd37868"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:21:54.960687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:54.960645    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-j5r4s\" (UniqueName: \"kubernetes.io/projected/25250034-7a4f-4fff-9efe-543efcd37868-kube-api-access-j5r4s\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:21:54.960687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:54.960682    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/25250034-7a4f-4fff-9efe-543efcd37868-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:21:55.595059 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:55.595031    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"
Nov 02 18:21:55.595224 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:55.595067    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn" event={"ID":"25250034-7a4f-4fff-9efe-543efcd37868","Type":"ContainerDied","Data":"d964146422313a4ba6f4f57331b55dacec6e17b11eb1f0611922ddb63d34c704"}
Nov 02 18:21:55.595224 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:21:55.595113    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="d964146422313a4ba6f4f57331b55dacec6e17b11eb1f0611922ddb63d34c704"
Nov 02 18:22:06.762871 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:06.762834    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:07.167091 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.167002    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"]
Nov 02 18:22:07.167456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.167441    4456 cpu_manager.go:414] "containerMap: removing container" podUID="25250034-7a4f-4fff-9efe-543efcd37868" containerName="revoke-keys"
Nov 02 18:22:07.167532 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.167457    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="25250034-7a4f-4fff-9efe-543efcd37868" containerName="revoke-keys"
Nov 02 18:22:07.167585 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.167535    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="25250034-7a4f-4fff-9efe-543efcd37868" containerName="revoke-keys"
Nov 02 18:22:07.170654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.170633    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.173355 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.173333    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-credential-socket\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.173450 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.173369    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-data\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.173491 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.173456    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-workload-certs\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.173526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.173517    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-workload-socket\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.173567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.173538    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-envoy\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.173622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.173610    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istiod-ca-cert\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.173656 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.173633    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-token\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.173689 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.173659    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mb86h\" (UniqueName: \"kubernetes.io/projected/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-kube-api-access-mb86h\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.173789 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.173764    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-podinfo\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.187586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.187566    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-ce839a-openshift-default-dockercfg-mjp2f\""
Nov 02 18:22:07.203967 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.203938    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"]
Nov 02 18:22:07.238230 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.238197    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:07.274240 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.274213    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-credential-socket\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.274392 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.274244    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-data\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.274392 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.274279    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-workload-certs\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.274392 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.274306    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-workload-socket\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.274392 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.274336    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-envoy\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.274586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.274365    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istiod-ca-cert\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.274586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.274380    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-token\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.274586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.274412    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mb86h\" (UniqueName: \"kubernetes.io/projected/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-kube-api-access-mb86h\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.274586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.274454    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-podinfo\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.274775 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.274704    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-data\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.274855 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.274829    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-workload-certs\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.274916 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.274865    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-credential-socket\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.274916 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.274907    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-workload-socket\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.275169 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.275144    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istiod-ca-cert\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.277102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.277077    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-envoy\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.277198 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.277181    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-podinfo\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.299266 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.299240    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mb86h\" (UniqueName: \"kubernetes.io/projected/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-kube-api-access-mb86h\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.299371 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.299282    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-token\") pod \"e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") " pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.380234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.380184    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:07.485229 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.485192    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:07.797447 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.797410    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"]
Nov 02 18:22:07.800542 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.800514    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:07.800629 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.800577    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:07.800629 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:07.800605    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:08.672367 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:08.672326    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9" event={"ID":"e7e6aec3-ddcc-4bed-b66a-b7e78259a972","Type":"ContainerStarted","Data":"4d2ea1c6a4e1f0ffb6fb40ceaf7811ef918f9c66fa702d4e012f88a41d18ec04"}
Nov 02 18:22:08.672575 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:08.672400    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9" event={"ID":"e7e6aec3-ddcc-4bed-b66a-b7e78259a972","Type":"ContainerStarted","Data":"c40322367ec818e997794489d3420bd28fff43da18ce787d572d05411a6a39c5"}
Nov 02 18:22:08.734712 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:08.734660    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9" podStartSLOduration=1.7346451 podStartE2EDuration="1.7346451s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:07 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:08.73151906 +0000 UTC m=+3145.222582220" watchObservedRunningTime="2026-11-02 18:22:08.7346451 +0000 UTC m=+3145.225708261"
Nov 02 18:22:09.485403 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:09.485365    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:09.490570 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:09.490544    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:09.678245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:09.678218    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:09.679265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:09.679245    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:11.260038 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:11.260001    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:11.417560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:11.417517    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:11.572165 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:11.572069    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:12.887956 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:12.887919    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"]
Nov 02 18:22:12.891946 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:12.891923    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:12.910668 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:12.910628    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 18:22:12.914985 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:12.914960    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"]
Nov 02 18:22:12.924068 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:12.924039    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-ce839a-serving-cert\""
Nov 02 18:22:12.924206 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:12.924145    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:22:12.924206 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:12.924174    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:22:12.931509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:12.931468    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:12.931635 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:12.931618    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:12.931700 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:12.931647    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-zn429\" (UniqueName: \"kubernetes.io/projected/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-kube-api-access-zn429\") pod \"maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:12.931700 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:12.931668    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:12.993051 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:12.993014    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:13.010670 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.010648    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gml5d\""
Nov 02 18:22:13.033249 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.033222    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:13.033341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.033261    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-zn429\" (UniqueName: \"kubernetes.io/projected/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-kube-api-access-zn429\") pod \"maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:13.033341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.033287    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:13.033341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.033333    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:13.033498 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:22:13.033479    4456 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-serving-cert: secret "maas-api-e2e-worker-single-main-ce839a-serving-cert" not found
Nov 02 18:22:13.033565 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:22:13.033553    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-maas-api-tls podName:860e2bd6-b26c-4e5d-8a79-3ca933fa8834 nodeName:}" failed. No retries permitted until 2026-11-02 18:22:13.533534179 +0000 UTC m=+3150.024597318 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-maas-api-tls") pod "maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c" (UID: "860e2bd6-b26c-4e5d-8a79-3ca933fa8834") : secret "maas-api-e2e-worker-single-main-ce839a-serving-cert" not found
Nov 02 18:22:13.033865 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.033834    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:13.036397 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.036314    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-metrics-certs\") pod \"maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:13.058944 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.058917    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-zn429\" (UniqueName: \"kubernetes.io/projected/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-kube-api-access-zn429\") pod \"maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:13.135693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.135660    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:13.280540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.280502    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:13.539009 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.538927    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:13.541643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.541617    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:13.812260 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.812161    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:13.949864 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.949804    4456 scope.go:122] "RemoveContainer" containerID="d78d07235afd10e0a4e730bf23edcf9f68d9d0481b60db6d00f9dbc678bacd64"
Nov 02 18:22:13.958718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.958691    4456 scope.go:122] "RemoveContainer" containerID="6676d84ebd251ade95948abf7c8d3e2f18afe551d6430fb4e11b2f461447d2f1"
Nov 02 18:22:13.966683 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:13.966663    4456 scope.go:122] "RemoveContainer" containerID="0694f718d8f4922ad5511fdd5cbacbe36d42c307f2281447e2e3eb26eb66fe86"
Nov 02 18:22:14.115266 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:14.115233    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"]
Nov 02 18:22:14.115866 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:22:14.115835    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod860e2bd6_b26c_4e5d_8a79_3ca933fa8834.slice/crio-18512dd2d3cd270df0802767486c274c0c45c593350d9a366885deca537da4a3 WatchSource:0}: Error finding container 18512dd2d3cd270df0802767486c274c0c45c593350d9a366885deca537da4a3: Status 404 returned error can't find the container with id 18512dd2d3cd270df0802767486c274c0c45c593350d9a366885deca537da4a3
Nov 02 18:22:14.117394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:14.117376    4456 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:22:14.711183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:14.711146    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c" event={"ID":"860e2bd6-b26c-4e5d-8a79-3ca933fa8834","Type":"ContainerStarted","Data":"b21c3f6993b4a49c88139346ba6fcd3b0822642602642706457055d9a62dec20"}
Nov 02 18:22:14.711362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:14.711206    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c" event={"ID":"860e2bd6-b26c-4e5d-8a79-3ca933fa8834","Type":"ContainerStarted","Data":"18512dd2d3cd270df0802767486c274c0c45c593350d9a366885deca537da4a3"}
Nov 02 18:22:14.711362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:14.711234    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:14.756551 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:14.756505    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c" podStartSLOduration=2.335258004 podStartE2EDuration="2.756489969s" totalImagesPullingTime="421.231965ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:22:12 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:22:14.755753145 +0000 UTC m=+3151.246816306" watchObservedRunningTime="2026-11-02 18:22:14.756489969 +0000 UTC m=+3151.247553129"
Nov 02 18:22:14.861024 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:14.860934    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:16.165898 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:22:16.165856    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:22:16.748651 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:16.748605    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:18.649587 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:18.649549    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:20.567804 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:20.567761    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:20.722161 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:20.722131    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:22:22.366237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:22.366203    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:24.270719 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:24.270681    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:25.719984 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:25.719947    4456 patch_prober.go:29] interesting pod/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:22:25.720465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:25.720010    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9" podUID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:22:26.069455 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:26.069358    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:26.216140 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:26.216104    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:26.763624 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:26.763597    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:26.780666 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:26.780641    4456 generic.go:356] "Generic (PLEG): container finished" podID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerID="4d2ea1c6a4e1f0ffb6fb40ceaf7811ef918f9c66fa702d4e012f88a41d18ec04" exitCode=0
Nov 02 18:22:26.780764 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:26.780693    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9" event={"ID":"e7e6aec3-ddcc-4bed-b66a-b7e78259a972","Type":"ContainerDied","Data":"4d2ea1c6a4e1f0ffb6fb40ceaf7811ef918f9c66fa702d4e012f88a41d18ec04"}
Nov 02 18:22:26.781071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:26.781057    4456 scope.go:122] "RemoveContainer" containerID="4d2ea1c6a4e1f0ffb6fb40ceaf7811ef918f9c66fa702d4e012f88a41d18ec04"
Nov 02 18:22:26.782225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:26.782201    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:26.782292 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:26.782266    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:26.782337 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:26.782298    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:22:26.917903 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:26.917755    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:27.058029 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:27.057926    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:27.810398 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:27.810361    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9" event={"ID":"e7e6aec3-ddcc-4bed-b66a-b7e78259a972","Type":"ContainerStarted","Data":"09ec63c67f5e4f14ae99c73f2efdd3fea87da2b8488be48c8314c646081ada19"}
Nov 02 18:22:27.958661 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:27.958626    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:28.401739 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:28.401701    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:28.493006 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:28.492967    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:28.493722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:28.493696    4456 patch_prober.go:29] interesting pod/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 02 18:22:28.493941 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:28.493918    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9" podUID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 02 18:22:29.489278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:29.489238    4456 patch_prober.go:29] interesting pod/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 02 18:22:29.489942 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:29.489901    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9" podUID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 02 18:22:29.837927 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:29.837806    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:30.486258 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:30.486219    4456 patch_prober.go:29] interesting pod/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 02 18:22:30.486550 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:30.486519    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9" podUID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 02 18:22:31.486246 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:31.486210    4456 patch_prober.go:29] interesting pod/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 02 18:22:31.487227 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:31.486842    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9" podUID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 02 18:22:31.765529 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:31.765449    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:32.486286 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:32.486247    4456 patch_prober.go:29] interesting pod/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9 container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused" start-of-body=
Nov 02 18:22:32.486929 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:32.486897    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9" podUID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.41:15021/healthz/ready\": dial tcp 10.134.1.41:15021: connect: connection refused"
Nov 02 18:22:33.489266 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:33.489237    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:33.489640 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:33.489502    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:33.490157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:33.490139    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:22:33.555608 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:33.555560    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:35.357647 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:35.357612    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:37.252135 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:37.252101    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:39.066791 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:39.066751    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:39.296280 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:39.296244    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:40.861655 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:40.861617    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:42.677247 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:42.677214    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:46.166848 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:22:46.166774    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:22:46.568718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:46.568685    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:46.803106 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:46.803072    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:48.158911 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:48.158877    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:49.950030 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:49.949996    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:51.759454 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:51.759412    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:53.546995 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:53.546953    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:55.349490 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:55.349455    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:57.147120 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:57.147083    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:22:58.960492 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:22:58.960432    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:00.945196 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:00.945165    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:02.740952 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:02.740913    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:04.559119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:04.559090    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:06.381469 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:06.381430    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:08.162372 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:08.162325    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:10.029209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:10.029166    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:11.660403 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:11.660328    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:11.869629 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:11.869594    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:13.891171 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:13.891126    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:14.032771 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:14.032739    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"]
Nov 02 18:23:14.039229 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:14.039196    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-2e1f5b5f-2gk4j"]
Nov 02 18:23:15.613521 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:15.613492    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7584f343-9a50-499d-a96b-132079baeff7" path="/var/lib/kubelet/pods/7584f343-9a50-499d-a96b-132079baeff7/volumes"
Nov 02 18:23:15.780772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:15.780738    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:16.165269 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:23:16.165232    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:23:16.687101 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:16.687065    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:17.648490 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:17.648436    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:19.538437 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:19.538406    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:21.351001 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:21.350964    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:22.498705 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:22.498674    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:22.644564 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:22.644522    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:22.833451 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:22.833360    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"]
Nov 02 18:23:22.833653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:22.833581    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c" podUID="860e2bd6-b26c-4e5d-8a79-3ca933fa8834" containerName="maas-api" containerID="cri-o://b21c3f6993b4a49c88139346ba6fcd3b0822642602642706457055d9a62dec20" gracePeriod=30
Nov 02 18:23:22.931035 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:22.930999    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:23.303186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:23.303162    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:23:23.470222 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:23.470185    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-metrics-certs\" (UniqueName: \"kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-metrics-certs\") pod \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") "
Nov 02 18:23:23.470441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:23.470257    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-kube-api-access-zn429\" (UniqueName: \"kubernetes.io/projected/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-kube-api-access-zn429\") pod \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") "
Nov 02 18:23:23.470441 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:23.470377    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-openshift-service-ca\") pod \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") "
Nov 02 18:23:23.470568 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:23.470544    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-maas-api-tls\") pod \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\" (UID: \"860e2bd6-b26c-4e5d-8a79-3ca933fa8834\") "
Nov 02 18:23:23.470723 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:23.470690    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-openshift-service-ca" pod "860e2bd6-b26c-4e5d-8a79-3ca933fa8834" (UID: "860e2bd6-b26c-4e5d-8a79-3ca933fa8834"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:23:23.471055 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:23.470981    4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-openshift-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:23.472732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:23.472706    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-metrics-certs" pod "860e2bd6-b26c-4e5d-8a79-3ca933fa8834" (UID: "860e2bd6-b26c-4e5d-8a79-3ca933fa8834"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:23:23.472843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:23.472739    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-maas-api-tls" pod "860e2bd6-b26c-4e5d-8a79-3ca933fa8834" (UID: "860e2bd6-b26c-4e5d-8a79-3ca933fa8834"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:23:23.472843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:23.472792    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-kube-api-access-zn429" pod "860e2bd6-b26c-4e5d-8a79-3ca933fa8834" (UID: "860e2bd6-b26c-4e5d-8a79-3ca933fa8834"). InnerVolumeSpecName "kube-api-access-zn429". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:23.571638 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:23.571610    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-maas-api-tls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:23.571638 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:23.571637    4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-metrics-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:23.571638 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:23.571646    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-zn429\" (UniqueName: \"kubernetes.io/projected/860e2bd6-b26c-4e5d-8a79-3ca933fa8834-kube-api-access-zn429\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:24.123780 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:24.123749    4456 generic.go:356] "Generic (PLEG): container finished" podID="860e2bd6-b26c-4e5d-8a79-3ca933fa8834" containerID="b21c3f6993b4a49c88139346ba6fcd3b0822642602642706457055d9a62dec20" exitCode=0
Nov 02 18:23:24.123971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:24.123794    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"
Nov 02 18:23:24.123971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:24.123809    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c" event={"ID":"860e2bd6-b26c-4e5d-8a79-3ca933fa8834","Type":"ContainerDied","Data":"b21c3f6993b4a49c88139346ba6fcd3b0822642602642706457055d9a62dec20"}
Nov 02 18:23:24.123971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:24.123868    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c" event={"ID":"860e2bd6-b26c-4e5d-8a79-3ca933fa8834","Type":"ContainerDied","Data":"18512dd2d3cd270df0802767486c274c0c45c593350d9a366885deca537da4a3"}
Nov 02 18:23:24.123971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:24.123888    4456 scope.go:122] "RemoveContainer" containerID="b21c3f6993b4a49c88139346ba6fcd3b0822642602642706457055d9a62dec20"
Nov 02 18:23:24.134158 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:24.134132    4456 scope.go:122] "RemoveContainer" containerID="b21c3f6993b4a49c88139346ba6fcd3b0822642602642706457055d9a62dec20"
Nov 02 18:23:24.134430 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:23:24.134409    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"b21c3f6993b4a49c88139346ba6fcd3b0822642602642706457055d9a62dec20\": container with ID starting with b21c3f6993b4a49c88139346ba6fcd3b0822642602642706457055d9a62dec20 not found: ID does not exist" containerID="b21c3f6993b4a49c88139346ba6fcd3b0822642602642706457055d9a62dec20"
Nov 02 18:23:24.134516 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:24.134445    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"b21c3f6993b4a49c88139346ba6fcd3b0822642602642706457055d9a62dec20"} err="failed to get container status \"b21c3f6993b4a49c88139346ba6fcd3b0822642602642706457055d9a62dec20\": rpc error: code = NotFound desc = could not find container \"b21c3f6993b4a49c88139346ba6fcd3b0822642602642706457055d9a62dec20\": container with ID starting with b21c3f6993b4a49c88139346ba6fcd3b0822642602642706457055d9a62dec20 not found: ID does not exist"
Nov 02 18:23:24.179687 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:24.179657    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"]
Nov 02 18:23:24.193091 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:24.193064    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-ce839a-754d68fcb9-bfj4c"]
Nov 02 18:23:25.042769 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:25.042727    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"]
Nov 02 18:23:25.055659 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:25.055620    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ec843018-72gr8"]
Nov 02 18:23:25.565442 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:25.565412    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:25.615517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:25.615479    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="860e2bd6-b26c-4e5d-8a79-3ca933fa8834" path="/var/lib/kubelet/pods/860e2bd6-b26c-4e5d-8a79-3ca933fa8834/volumes"
Nov 02 18:23:25.616138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:25.616110    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="88c20671-406b-4e7a-80fe-6d4a1b155980" path="/var/lib/kubelet/pods/88c20671-406b-4e7a-80fe-6d4a1b155980/volumes"
Nov 02 18:23:25.708017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:25.707983    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:26.227697 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:26.227660    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:32.422588 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:32.422550    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:32.567186 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:32.567143    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:32.733060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:32.733025    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"]
Nov 02 18:23:32.733307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:32.733279    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9" podUID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerName="istio-proxy" containerID="cri-o://09ec63c67f5e4f14ae99c73f2efdd3fea87da2b8488be48c8314c646081ada19" gracePeriod=30
Nov 02 18:23:32.801637 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:32.801609    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:33.492430 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:33.492397    4456 patch_prober.go:29] interesting pod/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9 container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:23:33.492800 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:33.492466    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9" podUID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:23:34.026612 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.026572    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"]
Nov 02 18:23:34.039679 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.039639    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-9kthn"]
Nov 02 18:23:34.078229 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.078194    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"]
Nov 02 18:23:34.078743 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.078723    4456 cpu_manager.go:414] "containerMap: removing container" podUID="860e2bd6-b26c-4e5d-8a79-3ca933fa8834" containerName="maas-api"
Nov 02 18:23:34.078743 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.078744    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="860e2bd6-b26c-4e5d-8a79-3ca933fa8834" containerName="maas-api"
Nov 02 18:23:34.078954 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.078865    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="860e2bd6-b26c-4e5d-8a79-3ca933fa8834" containerName="maas-api"
Nov 02 18:23:34.082124 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.082081    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"
Nov 02 18:23:34.104853 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.098544    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:23:34.104853 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.098642    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:23:34.104853 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.098698    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:23:34.104853 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.098545    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:23:34.104853 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.102289    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"]
Nov 02 18:23:34.180453 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.180421    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a336e820-a559-4d4f-b11a-c2d2da68254a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4krw5\" (UID: \"a336e820-a559-4d4f-b11a-c2d2da68254a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"
Nov 02 18:23:34.180631 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.180464    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6nlbf\" (UniqueName: \"kubernetes.io/projected/a336e820-a559-4d4f-b11a-c2d2da68254a-kube-api-access-6nlbf\") pod \"maas-api-revoke-sub-3d4156f1-4krw5\" (UID: \"a336e820-a559-4d4f-b11a-c2d2da68254a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"
Nov 02 18:23:34.180631 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.180487    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a336e820-a559-4d4f-b11a-c2d2da68254a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4krw5\" (UID: \"a336e820-a559-4d4f-b11a-c2d2da68254a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"
Nov 02 18:23:34.281642 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.281566    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a336e820-a559-4d4f-b11a-c2d2da68254a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4krw5\" (UID: \"a336e820-a559-4d4f-b11a-c2d2da68254a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"
Nov 02 18:23:34.281642 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.281610    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6nlbf\" (UniqueName: \"kubernetes.io/projected/a336e820-a559-4d4f-b11a-c2d2da68254a-kube-api-access-6nlbf\") pod \"maas-api-revoke-sub-3d4156f1-4krw5\" (UID: \"a336e820-a559-4d4f-b11a-c2d2da68254a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"
Nov 02 18:23:34.281863 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.281753    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a336e820-a559-4d4f-b11a-c2d2da68254a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4krw5\" (UID: \"a336e820-a559-4d4f-b11a-c2d2da68254a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"
Nov 02 18:23:34.282292 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.282270    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a336e820-a559-4d4f-b11a-c2d2da68254a-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3d4156f1-4krw5\" (UID: \"a336e820-a559-4d4f-b11a-c2d2da68254a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"
Nov 02 18:23:34.284277 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.284257    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a336e820-a559-4d4f-b11a-c2d2da68254a-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3d4156f1-4krw5\" (UID: \"a336e820-a559-4d4f-b11a-c2d2da68254a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"
Nov 02 18:23:34.306117 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.306090    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6nlbf\" (UniqueName: \"kubernetes.io/projected/a336e820-a559-4d4f-b11a-c2d2da68254a-kube-api-access-6nlbf\") pod \"maas-api-revoke-sub-3d4156f1-4krw5\" (UID: \"a336e820-a559-4d4f-b11a-c2d2da68254a\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"
Nov 02 18:23:34.395086 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.395055    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"
Nov 02 18:23:34.712538 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:23:34.712504    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda336e820_a559_4d4f_b11a_c2d2da68254a.slice/crio-1852dc2ad51edb203a865df12fe2c8e75f890f1b39bebf2cd41f98de4d027bd9 WatchSource:0}: Error finding container 1852dc2ad51edb203a865df12fe2c8e75f890f1b39bebf2cd41f98de4d027bd9: Status 404 returned error can't find the container with id 1852dc2ad51edb203a865df12fe2c8e75f890f1b39bebf2cd41f98de4d027bd9
Nov 02 18:23:34.713766 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:34.713735    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"]
Nov 02 18:23:35.188447 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:35.188409    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5" event={"ID":"a336e820-a559-4d4f-b11a-c2d2da68254a","Type":"ContainerStarted","Data":"240a29cc3ce6c00cbe75b862d47d92e5e891e23abaa4fa7ea261b20767c9569c"}
Nov 02 18:23:35.188611 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:35.188467    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5" event={"ID":"a336e820-a559-4d4f-b11a-c2d2da68254a","Type":"ContainerStarted","Data":"1852dc2ad51edb203a865df12fe2c8e75f890f1b39bebf2cd41f98de4d027bd9"}
Nov 02 18:23:35.232388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:35.232342    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5" podStartSLOduration=1.232329077 podStartE2EDuration="1.232329077s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:23:34 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:23:35.231116591 +0000 UTC m=+3231.722179762" watchObservedRunningTime="2026-11-02 18:23:35.232329077 +0000 UTC m=+3231.723392238"
Nov 02 18:23:35.613732 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:35.613707    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="25250034-7a4f-4fff-9efe-543efcd37868" path="/var/lib/kubelet/pods/25250034-7a4f-4fff-9efe-543efcd37868/volumes"
Nov 02 18:23:36.195365 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:36.195329    4456 generic.go:356] "Generic (PLEG): container finished" podID="a336e820-a559-4d4f-b11a-c2d2da68254a" containerID="240a29cc3ce6c00cbe75b862d47d92e5e891e23abaa4fa7ea261b20767c9569c" exitCode=0
Nov 02 18:23:36.195765 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:36.195413    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5" event={"ID":"a336e820-a559-4d4f-b11a-c2d2da68254a","Type":"ContainerDied","Data":"240a29cc3ce6c00cbe75b862d47d92e5e891e23abaa4fa7ea261b20767c9569c"}
Nov 02 18:23:37.339661 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:37.339635    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"
Nov 02 18:23:37.412761 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:37.412733    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a336e820-a559-4d4f-b11a-c2d2da68254a-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a336e820-a559-4d4f-b11a-c2d2da68254a-maas-api-openshift-ca\") pod \"a336e820-a559-4d4f-b11a-c2d2da68254a\" (UID: \"a336e820-a559-4d4f-b11a-c2d2da68254a\") "
Nov 02 18:23:37.412935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:37.412918    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a336e820-a559-4d4f-b11a-c2d2da68254a-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a336e820-a559-4d4f-b11a-c2d2da68254a-maas-api-xks-ca\") pod \"a336e820-a559-4d4f-b11a-c2d2da68254a\" (UID: \"a336e820-a559-4d4f-b11a-c2d2da68254a\") "
Nov 02 18:23:37.412979 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:37.412958    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a336e820-a559-4d4f-b11a-c2d2da68254a-kube-api-access-6nlbf\" (UniqueName: \"kubernetes.io/projected/a336e820-a559-4d4f-b11a-c2d2da68254a-kube-api-access-6nlbf\") pod \"a336e820-a559-4d4f-b11a-c2d2da68254a\" (UID: \"a336e820-a559-4d4f-b11a-c2d2da68254a\") "
Nov 02 18:23:37.413093 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:37.413062    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a336e820-a559-4d4f-b11a-c2d2da68254a-maas-api-openshift-ca" pod "a336e820-a559-4d4f-b11a-c2d2da68254a" (UID: "a336e820-a559-4d4f-b11a-c2d2da68254a"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:23:37.413259 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:37.413239    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/a336e820-a559-4d4f-b11a-c2d2da68254a-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:37.418150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:37.418125    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a336e820-a559-4d4f-b11a-c2d2da68254a-maas-api-xks-ca" pod "a336e820-a559-4d4f-b11a-c2d2da68254a" (UID: "a336e820-a559-4d4f-b11a-c2d2da68254a"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:23:37.418269 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:37.418197    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a336e820-a559-4d4f-b11a-c2d2da68254a-kube-api-access-6nlbf" pod "a336e820-a559-4d4f-b11a-c2d2da68254a" (UID: "a336e820-a559-4d4f-b11a-c2d2da68254a"). InnerVolumeSpecName "kube-api-access-6nlbf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:37.513857 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:37.513758    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/a336e820-a559-4d4f-b11a-c2d2da68254a-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:37.513857 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:37.513785    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6nlbf\" (UniqueName: \"kubernetes.io/projected/a336e820-a559-4d4f-b11a-c2d2da68254a-kube-api-access-6nlbf\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:37.971446 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:37.971419    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:23:38.018318 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.018293    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-envoy\") pod \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") "
Nov 02 18:23:38.018460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.018353    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-kube-api-access-mb86h\" (UniqueName: \"kubernetes.io/projected/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-kube-api-access-mb86h\") pod \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") "
Nov 02 18:23:38.018460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.018386    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istiod-ca-cert\") pod \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") "
Nov 02 18:23:38.018460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.018426    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-workload-certs\") pod \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") "
Nov 02 18:23:38.018614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.018464    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-data\") pod \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") "
Nov 02 18:23:38.018614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.018492    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-workload-socket\") pod \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") "
Nov 02 18:23:38.018614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.018514    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-token\" (UniqueName: \"kubernetes.io/projected/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-token\") pod \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") "
Nov 02 18:23:38.018614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.018557    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-podinfo\") pod \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") "
Nov 02 18:23:38.018614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.018585    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-credential-socket\") pod \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\" (UID: \"e7e6aec3-ddcc-4bed-b66a-b7e78259a972\") "
Nov 02 18:23:38.018905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.018712    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-workload-certs" pod "e7e6aec3-ddcc-4bed-b66a-b7e78259a972" (UID: "e7e6aec3-ddcc-4bed-b66a-b7e78259a972"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:38.018905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.018810    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-data" pod "e7e6aec3-ddcc-4bed-b66a-b7e78259a972" (UID: "e7e6aec3-ddcc-4bed-b66a-b7e78259a972"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:38.018905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.018886    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istiod-ca-cert" pod "e7e6aec3-ddcc-4bed-b66a-b7e78259a972" (UID: "e7e6aec3-ddcc-4bed-b66a-b7e78259a972"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:23:38.019059 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.018984    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:38.019059 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.019038    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:38.019059 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.019051    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:38.019218 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.019075    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-credential-socket" pod "e7e6aec3-ddcc-4bed-b66a-b7e78259a972" (UID: "e7e6aec3-ddcc-4bed-b66a-b7e78259a972"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:38.019346 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.019325    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-workload-socket" pod "e7e6aec3-ddcc-4bed-b66a-b7e78259a972" (UID: "e7e6aec3-ddcc-4bed-b66a-b7e78259a972"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:38.020912 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.020888    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-envoy" pod "e7e6aec3-ddcc-4bed-b66a-b7e78259a972" (UID: "e7e6aec3-ddcc-4bed-b66a-b7e78259a972"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:38.021030 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.020945    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-podinfo" pod "e7e6aec3-ddcc-4bed-b66a-b7e78259a972" (UID: "e7e6aec3-ddcc-4bed-b66a-b7e78259a972"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:23:38.021075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.021045    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-kube-api-access-mb86h" pod "e7e6aec3-ddcc-4bed-b66a-b7e78259a972" (UID: "e7e6aec3-ddcc-4bed-b66a-b7e78259a972"). InnerVolumeSpecName "kube-api-access-mb86h". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:38.021233 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.021215    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-token" pod "e7e6aec3-ddcc-4bed-b66a-b7e78259a972" (UID: "e7e6aec3-ddcc-4bed-b66a-b7e78259a972"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:38.119622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.119544    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:38.119622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.119571    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:38.119622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.119581    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:38.119622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.119590    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mb86h\" (UniqueName: \"kubernetes.io/projected/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-kube-api-access-mb86h\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:38.119622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.119603    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:38.119622 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.119616    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/e7e6aec3-ddcc-4bed-b66a-b7e78259a972-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:38.207967 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.207937    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"
Nov 02 18:23:38.208142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.207983    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5" event={"ID":"a336e820-a559-4d4f-b11a-c2d2da68254a","Type":"ContainerDied","Data":"1852dc2ad51edb203a865df12fe2c8e75f890f1b39bebf2cd41f98de4d027bd9"}
Nov 02 18:23:38.208142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.208066    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="1852dc2ad51edb203a865df12fe2c8e75f890f1b39bebf2cd41f98de4d027bd9"
Nov 02 18:23:38.210084 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.210053    4456 generic.go:356] "Generic (PLEG): container finished" podID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerID="09ec63c67f5e4f14ae99c73f2efdd3fea87da2b8488be48c8314c646081ada19" exitCode=0
Nov 02 18:23:38.210229 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.210132    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9" event={"ID":"e7e6aec3-ddcc-4bed-b66a-b7e78259a972","Type":"ContainerDied","Data":"09ec63c67f5e4f14ae99c73f2efdd3fea87da2b8488be48c8314c646081ada19"}
Nov 02 18:23:38.210229 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.210176    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9" event={"ID":"e7e6aec3-ddcc-4bed-b66a-b7e78259a972","Type":"ContainerDied","Data":"c40322367ec818e997794489d3420bd28fff43da18ce787d572d05411a6a39c5"}
Nov 02 18:23:38.210229 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.210205    4456 scope.go:122] "RemoveContainer" containerID="09ec63c67f5e4f14ae99c73f2efdd3fea87da2b8488be48c8314c646081ada19"
Nov 02 18:23:38.210405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.210368    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"
Nov 02 18:23:38.223678 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.223658    4456 scope.go:122] "RemoveContainer" containerID="4d2ea1c6a4e1f0ffb6fb40ceaf7811ef918f9c66fa702d4e012f88a41d18ec04"
Nov 02 18:23:38.233114 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.233099    4456 scope.go:122] "RemoveContainer" containerID="09ec63c67f5e4f14ae99c73f2efdd3fea87da2b8488be48c8314c646081ada19"
Nov 02 18:23:38.233370 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:23:38.233349    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"09ec63c67f5e4f14ae99c73f2efdd3fea87da2b8488be48c8314c646081ada19\": container with ID starting with 09ec63c67f5e4f14ae99c73f2efdd3fea87da2b8488be48c8314c646081ada19 not found: ID does not exist" containerID="09ec63c67f5e4f14ae99c73f2efdd3fea87da2b8488be48c8314c646081ada19"
Nov 02 18:23:38.233431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.233376    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"09ec63c67f5e4f14ae99c73f2efdd3fea87da2b8488be48c8314c646081ada19"} err="failed to get container status \"09ec63c67f5e4f14ae99c73f2efdd3fea87da2b8488be48c8314c646081ada19\": rpc error: code = NotFound desc = could not find container \"09ec63c67f5e4f14ae99c73f2efdd3fea87da2b8488be48c8314c646081ada19\": container with ID starting with 09ec63c67f5e4f14ae99c73f2efdd3fea87da2b8488be48c8314c646081ada19 not found: ID does not exist"
Nov 02 18:23:38.233431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.233392    4456 scope.go:122] "RemoveContainer" containerID="4d2ea1c6a4e1f0ffb6fb40ceaf7811ef918f9c66fa702d4e012f88a41d18ec04"
Nov 02 18:23:38.233638 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:23:38.233624    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4d2ea1c6a4e1f0ffb6fb40ceaf7811ef918f9c66fa702d4e012f88a41d18ec04\": container with ID starting with 4d2ea1c6a4e1f0ffb6fb40ceaf7811ef918f9c66fa702d4e012f88a41d18ec04 not found: ID does not exist" containerID="4d2ea1c6a4e1f0ffb6fb40ceaf7811ef918f9c66fa702d4e012f88a41d18ec04"
Nov 02 18:23:38.233699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.233641    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4d2ea1c6a4e1f0ffb6fb40ceaf7811ef918f9c66fa702d4e012f88a41d18ec04"} err="failed to get container status \"4d2ea1c6a4e1f0ffb6fb40ceaf7811ef918f9c66fa702d4e012f88a41d18ec04\": rpc error: code = NotFound desc = could not find container \"4d2ea1c6a4e1f0ffb6fb40ceaf7811ef918f9c66fa702d4e012f88a41d18ec04\": container with ID starting with 4d2ea1c6a4e1f0ffb6fb40ceaf7811ef918f9c66fa702d4e012f88a41d18ec04 not found: ID does not exist"
Nov 02 18:23:38.288592 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.288568    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"]
Nov 02 18:23:38.292381 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:38.292360    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-ce839a-openshift-default-597b95d789mb2c9"]
Nov 02 18:23:39.410302 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:39.410272    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:39.612583 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:39.612546    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" path="/var/lib/kubelet/pods/e7e6aec3-ddcc-4bed-b66a-b7e78259a972/volumes"
Nov 02 18:23:46.166554 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:23:46.166517    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:23:51.954510 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:51.954474    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"]
Nov 02 18:23:51.954931 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:51.954676    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager" containerID="cri-o://4a523413662dfad5f79a825bb0d5c61b2430be82132b00a3deb4475053a2e2c9" gracePeriod=10
Nov 02 18:23:52.031860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.031794    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"]
Nov 02 18:23:52.034780 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.034747    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-046e1343-kz6jz"]
Nov 02 18:23:52.211389 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.211337    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 18:23:52.288974 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.288939    4456 generic.go:356] "Generic (PLEG): container finished" podID="a1ea3dac-7073-4564-b247-89ea94a64342" containerID="4a523413662dfad5f79a825bb0d5c61b2430be82132b00a3deb4475053a2e2c9" exitCode=0
Nov 02 18:23:52.289156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.288993    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"
Nov 02 18:23:52.289156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.289011    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" event={"ID":"a1ea3dac-7073-4564-b247-89ea94a64342","Type":"ContainerDied","Data":"4a523413662dfad5f79a825bb0d5c61b2430be82132b00a3deb4475053a2e2c9"}
Nov 02 18:23:52.289156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.289095    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8" event={"ID":"a1ea3dac-7073-4564-b247-89ea94a64342","Type":"ContainerDied","Data":"46103e666e23a14578de5271d8da8f28533195c6f4d5e5ef5cdbdc44d4803cf3"}
Nov 02 18:23:52.289156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.289128    4456 scope.go:122] "RemoveContainer" containerID="4a523413662dfad5f79a825bb0d5c61b2430be82132b00a3deb4475053a2e2c9"
Nov 02 18:23:52.303997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.303977    4456 scope.go:122] "RemoveContainer" containerID="f46a45f18a610493bf46d8787ee563938a877ab2e179a2e516f44f6e70fcc573"
Nov 02 18:23:52.315577 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.315557    4456 scope.go:122] "RemoveContainer" containerID="4a523413662dfad5f79a825bb0d5c61b2430be82132b00a3deb4475053a2e2c9"
Nov 02 18:23:52.315959 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:23:52.315934    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"4a523413662dfad5f79a825bb0d5c61b2430be82132b00a3deb4475053a2e2c9\": container with ID starting with 4a523413662dfad5f79a825bb0d5c61b2430be82132b00a3deb4475053a2e2c9 not found: ID does not exist" containerID="4a523413662dfad5f79a825bb0d5c61b2430be82132b00a3deb4475053a2e2c9"
Nov 02 18:23:52.316063 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.315968    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"4a523413662dfad5f79a825bb0d5c61b2430be82132b00a3deb4475053a2e2c9"} err="failed to get container status \"4a523413662dfad5f79a825bb0d5c61b2430be82132b00a3deb4475053a2e2c9\": rpc error: code = NotFound desc = could not find container \"4a523413662dfad5f79a825bb0d5c61b2430be82132b00a3deb4475053a2e2c9\": container with ID starting with 4a523413662dfad5f79a825bb0d5c61b2430be82132b00a3deb4475053a2e2c9 not found: ID does not exist"
Nov 02 18:23:52.316063 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.315987    4456 scope.go:122] "RemoveContainer" containerID="f46a45f18a610493bf46d8787ee563938a877ab2e179a2e516f44f6e70fcc573"
Nov 02 18:23:52.316292 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:23:52.316266    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"f46a45f18a610493bf46d8787ee563938a877ab2e179a2e516f44f6e70fcc573\": container with ID starting with f46a45f18a610493bf46d8787ee563938a877ab2e179a2e516f44f6e70fcc573 not found: ID does not exist" containerID="f46a45f18a610493bf46d8787ee563938a877ab2e179a2e516f44f6e70fcc573"
Nov 02 18:23:52.316357 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.316298    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"f46a45f18a610493bf46d8787ee563938a877ab2e179a2e516f44f6e70fcc573"} err="failed to get container status \"f46a45f18a610493bf46d8787ee563938a877ab2e179a2e516f44f6e70fcc573\": rpc error: code = NotFound desc = could not find container \"f46a45f18a610493bf46d8787ee563938a877ab2e179a2e516f44f6e70fcc573\": container with ID starting with f46a45f18a610493bf46d8787ee563938a877ab2e179a2e516f44f6e70fcc573 not found: ID does not exist"
Nov 02 18:23:52.356336 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.356310    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a1ea3dac-7073-4564-b247-89ea94a64342-kube-api-access-s5bh4\" (UniqueName: \"kubernetes.io/projected/a1ea3dac-7073-4564-b247-89ea94a64342-kube-api-access-s5bh4\") pod \"a1ea3dac-7073-4564-b247-89ea94a64342\" (UID: \"a1ea3dac-7073-4564-b247-89ea94a64342\") "
Nov 02 18:23:52.356465 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.356402    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/a1ea3dac-7073-4564-b247-89ea94a64342-extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1ea3dac-7073-4564-b247-89ea94a64342-extensions-socket-volume\") pod \"a1ea3dac-7073-4564-b247-89ea94a64342\" (UID: \"a1ea3dac-7073-4564-b247-89ea94a64342\") "
Nov 02 18:23:52.357283 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.357261    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/a1ea3dac-7073-4564-b247-89ea94a64342-extensions-socket-volume" pod "a1ea3dac-7073-4564-b247-89ea94a64342" (UID: "a1ea3dac-7073-4564-b247-89ea94a64342"). InnerVolumeSpecName "extensions-socket-volume". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:23:52.359570 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.359543    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a1ea3dac-7073-4564-b247-89ea94a64342-kube-api-access-s5bh4" pod "a1ea3dac-7073-4564-b247-89ea94a64342" (UID: "a1ea3dac-7073-4564-b247-89ea94a64342"). InnerVolumeSpecName "kube-api-access-s5bh4". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:23:52.457971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.457940    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-s5bh4\" (UniqueName: \"kubernetes.io/projected/a1ea3dac-7073-4564-b247-89ea94a64342-kube-api-access-s5bh4\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:52.457971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.457966    4456 reconciler_common.go:299] "Volume detached for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/a1ea3dac-7073-4564-b247-89ea94a64342-extensions-socket-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:23:52.638842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.638796    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"]
Nov 02 18:23:52.644513 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:52.644492    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-6jpq8"]
Nov 02 18:23:53.612163 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:53.612132    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="73cfd712-6978-4e5b-a5a2-1641533d46d4" path="/var/lib/kubelet/pods/73cfd712-6978-4e5b-a5a2-1641533d46d4/volumes"
Nov 02 18:23:53.612624 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:53.612607    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" path="/var/lib/kubelet/pods/a1ea3dac-7073-4564-b247-89ea94a64342/volumes"
Nov 02 18:23:53.922873 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:53.922762    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:54.226703 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:54.226672    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:23:59.905404 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:23:59.905372    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:01.007864 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:01.007831    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:02.050532 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:02.050498    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"]
Nov 02 18:24:02.063200 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:02.063173    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d3fba056-vtwgc"]
Nov 02 18:24:03.612871 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:03.612842    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="dccc5fbb-b40d-4e99-8195-24cb2440c21a" path="/var/lib/kubelet/pods/dccc5fbb-b40d-4e99-8195-24cb2440c21a/volumes"
Nov 02 18:24:14.007085 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:14.007052    4456 scope.go:122] "RemoveContainer" containerID="aabae281835c289c50c96a5e221bc132e31fb1da3a3358cdcf310c506d5aff37"
Nov 02 18:24:16.164622 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:24:16.164587    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:24:23.035217 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:23.035184    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"]
Nov 02 18:24:23.047682 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:23.047657    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-ae594979-v5xqz"]
Nov 02 18:24:23.612722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:23.612688    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d07e279e-9e65-43b9-8a3e-ea3927f0bf5d" path="/var/lib/kubelet/pods/d07e279e-9e65-43b9-8a3e-ea3927f0bf5d/volumes"
Nov 02 18:24:39.538089 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:39.538042    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:42.792600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:42.792512    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:45.032932 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:45.032884    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"]
Nov 02 18:24:45.045831 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:45.045796    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-a4aef7d7-zbqlf"]
Nov 02 18:24:45.231329 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:45.231279    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:24:45.612381 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:45.612346    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="007a9ed2-f45f-4cd9-b033-2e1ce58088ca" path="/var/lib/kubelet/pods/007a9ed2-f45f-4cd9-b033-2e1ce58088ca/volumes"
Nov 02 18:24:46.164084 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:24:46.164047    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:24:55.098620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.098585    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf"]
Nov 02 18:24:55.099132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.099012    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerName="istio-proxy"
Nov 02 18:24:55.099132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.099024    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerName="istio-proxy"
Nov 02 18:24:55.099132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.099039    4456 cpu_manager.go:414] "containerMap: removing container" podUID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerName="istio-proxy"
Nov 02 18:24:55.099132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.099045    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerName="istio-proxy"
Nov 02 18:24:55.099132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.099053    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager"
Nov 02 18:24:55.099132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.099057    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager"
Nov 02 18:24:55.099132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.099066    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a336e820-a559-4d4f-b11a-c2d2da68254a" containerName="revoke-keys"
Nov 02 18:24:55.099132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.099071    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a336e820-a559-4d4f-b11a-c2d2da68254a" containerName="revoke-keys"
Nov 02 18:24:55.099132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.099088    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager"
Nov 02 18:24:55.099132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.099092    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager"
Nov 02 18:24:55.099431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.099162    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerName="istio-proxy"
Nov 02 18:24:55.099431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.099172    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="e7e6aec3-ddcc-4bed-b66a-b7e78259a972" containerName="istio-proxy"
Nov 02 18:24:55.099431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.099178    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager"
Nov 02 18:24:55.099431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.099183    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a336e820-a559-4d4f-b11a-c2d2da68254a" containerName="revoke-keys"
Nov 02 18:24:55.099431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.099193    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager"
Nov 02 18:24:55.101212 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.101195    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf"
Nov 02 18:24:55.117041 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.117009    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf"]
Nov 02 18:24:55.128026 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.127996    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"kuadrant-system\"/\"kuadrant-operator-controller-manager-dockercfg-p2jm6\""
Nov 02 18:24:55.229803 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.229772    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39b9a295-67cb-402a-9f11-41d0e8e7d2eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-47dcf\" (UID: \"39b9a295-67cb-402a-9f11-41d0e8e7d2eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf"
Nov 02 18:24:55.229997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.229931    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-bpmnn\" (UniqueName: \"kubernetes.io/projected/39b9a295-67cb-402a-9f11-41d0e8e7d2eb-kube-api-access-bpmnn\") pod \"kuadrant-operator-controller-manager-678d87766f-47dcf\" (UID: \"39b9a295-67cb-402a-9f11-41d0e8e7d2eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf"
Nov 02 18:24:55.330682 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.330645    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39b9a295-67cb-402a-9f11-41d0e8e7d2eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-47dcf\" (UID: \"39b9a295-67cb-402a-9f11-41d0e8e7d2eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf"
Nov 02 18:24:55.330884 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.330723    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-bpmnn\" (UniqueName: \"kubernetes.io/projected/39b9a295-67cb-402a-9f11-41d0e8e7d2eb-kube-api-access-bpmnn\") pod \"kuadrant-operator-controller-manager-678d87766f-47dcf\" (UID: \"39b9a295-67cb-402a-9f11-41d0e8e7d2eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf"
Nov 02 18:24:55.331150 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.331125    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"extensions-socket-volume\" (UniqueName: \"kubernetes.io/empty-dir/39b9a295-67cb-402a-9f11-41d0e8e7d2eb-extensions-socket-volume\") pod \"kuadrant-operator-controller-manager-678d87766f-47dcf\" (UID: \"39b9a295-67cb-402a-9f11-41d0e8e7d2eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf"
Nov 02 18:24:55.355597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.355531    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-bpmnn\" (UniqueName: \"kubernetes.io/projected/39b9a295-67cb-402a-9f11-41d0e8e7d2eb-kube-api-access-bpmnn\") pod \"kuadrant-operator-controller-manager-678d87766f-47dcf\" (UID: \"39b9a295-67cb-402a-9f11-41d0e8e7d2eb\") " pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf"
Nov 02 18:24:55.413264 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.413230    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf"
Nov 02 18:24:55.736588 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:55.736560    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf"]
Nov 02 18:24:55.737827 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:24:55.737788    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod39b9a295_67cb_402a_9f11_41d0e8e7d2eb.slice/crio-f0f86d0378f0949bf2c18e7d4b8610f388795379c044c0712eaf874938361f00 WatchSource:0}: Error finding container f0f86d0378f0949bf2c18e7d4b8610f388795379c044c0712eaf874938361f00: Status 404 returned error can't find the container with id f0f86d0378f0949bf2c18e7d4b8610f388795379c044c0712eaf874938361f00
Nov 02 18:24:56.619832 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:56.619780    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf" event={"ID":"39b9a295-67cb-402a-9f11-41d0e8e7d2eb","Type":"ContainerStarted","Data":"2695ed134fb84b173af39fa08c1683588b98bb8fef48602b122e67c734cf2c9e"}
Nov 02 18:24:56.620245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:56.619846    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf" event={"ID":"39b9a295-67cb-402a-9f11-41d0e8e7d2eb","Type":"ContainerStarted","Data":"f0f86d0378f0949bf2c18e7d4b8610f388795379c044c0712eaf874938361f00"}
Nov 02 18:24:56.620245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:56.619941    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf"
Nov 02 18:24:56.694535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:24:56.694489    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf" podStartSLOduration=1.694474455 podStartE2EDuration="1.694474455s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:24:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:24:56.693016962 +0000 UTC m=+3313.184080122" watchObservedRunningTime="2026-11-02 18:24:56.694474455 +0000 UTC m=+3313.185537616"
Nov 02 18:25:06.030167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:06.030133    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"]
Nov 02 18:25:06.042901 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:06.042874    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-79805ca9-6fhn9"]
Nov 02 18:25:07.029332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:07.029291    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:07.172562 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:07.172527    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:07.618400 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:07.618354    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="fda269f0-9046-47e3-ae35-c4469ddf41f7" path="/var/lib/kubelet/pods/fda269f0-9046-47e3-ae35-c4469ddf41f7/volumes"
Nov 02 18:25:07.626347 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:07.626316    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="kuadrant-system/kuadrant-operator-controller-manager-678d87766f-47dcf"
Nov 02 18:25:14.079675 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:14.079639    4456 scope.go:122] "RemoveContainer" containerID="6f00e4f1b990da667fb5172c8ce252417cd8e6f9a40b43ffd2b4693c83afaa3e"
Nov 02 18:25:14.088511 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:14.088490    4456 scope.go:122] "RemoveContainer" containerID="82a39ba47d98ca26c5525d06cb177da7d0e1c2cc2bba443b377e3ed46d18afc5"
Nov 02 18:25:14.096545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:14.096525    4456 scope.go:122] "RemoveContainer" containerID="8d3998bd52036ca16be6d5cd29056cdd91c2e28594d07f9b89ebce19e09bba4c"
Nov 02 18:25:16.178755 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:25:16.178721    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:25:18.863849 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:18.863805    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"]
Nov 02 18:25:18.864308 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:18.864253    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager"
Nov 02 18:25:18.864308 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:18.864266    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager"
Nov 02 18:25:18.864378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:18.864336    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a1ea3dac-7073-4564-b247-89ea94a64342" containerName="manager"
Nov 02 18:25:18.866786 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:18.866768    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"
Nov 02 18:25:18.887266 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:18.887225    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:25:18.898034 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:18.898010    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:25:18.898193 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:18.898021    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:25:18.898252 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:18.898054    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:25:18.900425 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:18.900405    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"]
Nov 02 18:25:18.986157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:18.986094    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-6n4cr\" (UniqueName: \"kubernetes.io/projected/13f6ef0d-fbff-49c2-b187-d4340496ee8b-kube-api-access-6n4cr\") pod \"maas-api-revoke-sub-d576e68b-rtv5w\" (UID: \"13f6ef0d-fbff-49c2-b187-d4340496ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"
Nov 02 18:25:18.986157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:18.986164    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f6ef0d-fbff-49c2-b187-d4340496ee8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-rtv5w\" (UID: \"13f6ef0d-fbff-49c2-b187-d4340496ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"
Nov 02 18:25:18.986410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:18.986318    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f6ef0d-fbff-49c2-b187-d4340496ee8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-rtv5w\" (UID: \"13f6ef0d-fbff-49c2-b187-d4340496ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"
Nov 02 18:25:19.087618 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:19.087570    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f6ef0d-fbff-49c2-b187-d4340496ee8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-rtv5w\" (UID: \"13f6ef0d-fbff-49c2-b187-d4340496ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"
Nov 02 18:25:19.087809 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:19.087646    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-6n4cr\" (UniqueName: \"kubernetes.io/projected/13f6ef0d-fbff-49c2-b187-d4340496ee8b-kube-api-access-6n4cr\") pod \"maas-api-revoke-sub-d576e68b-rtv5w\" (UID: \"13f6ef0d-fbff-49c2-b187-d4340496ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"
Nov 02 18:25:19.087809 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:19.087717    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f6ef0d-fbff-49c2-b187-d4340496ee8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-rtv5w\" (UID: \"13f6ef0d-fbff-49c2-b187-d4340496ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"
Nov 02 18:25:19.088512 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:19.088483    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f6ef0d-fbff-49c2-b187-d4340496ee8b-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d576e68b-rtv5w\" (UID: \"13f6ef0d-fbff-49c2-b187-d4340496ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"
Nov 02 18:25:19.090124 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:19.090099    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f6ef0d-fbff-49c2-b187-d4340496ee8b-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d576e68b-rtv5w\" (UID: \"13f6ef0d-fbff-49c2-b187-d4340496ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"
Nov 02 18:25:19.112625 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:19.112588    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-6n4cr\" (UniqueName: \"kubernetes.io/projected/13f6ef0d-fbff-49c2-b187-d4340496ee8b-kube-api-access-6n4cr\") pod \"maas-api-revoke-sub-d576e68b-rtv5w\" (UID: \"13f6ef0d-fbff-49c2-b187-d4340496ee8b\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"
Nov 02 18:25:19.179224 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:19.179158    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"
Nov 02 18:25:19.481345 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:19.481316    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"]
Nov 02 18:25:19.482359 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:25:19.482323    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod13f6ef0d_fbff_49c2_b187_d4340496ee8b.slice/crio-8fc51ef50010664cb138c9511f23b6e2cc3e440047c95345c4e24119495bd4b6 WatchSource:0}: Error finding container 8fc51ef50010664cb138c9511f23b6e2cc3e440047c95345c4e24119495bd4b6: Status 404 returned error can't find the container with id 8fc51ef50010664cb138c9511f23b6e2cc3e440047c95345c4e24119495bd4b6
Nov 02 18:25:19.751760 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:19.751691    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w" event={"ID":"13f6ef0d-fbff-49c2-b187-d4340496ee8b","Type":"ContainerStarted","Data":"8fc51ef50010664cb138c9511f23b6e2cc3e440047c95345c4e24119495bd4b6"}
Nov 02 18:25:20.766470 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:20.766432    4456 generic.go:356] "Generic (PLEG): container finished" podID="13f6ef0d-fbff-49c2-b187-d4340496ee8b" containerID="d793c2237d65c37419b1d637f39a881b7ba53dbb7ddc89c82d94a62d448439ec" exitCode=0
Nov 02 18:25:20.766878 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:20.766519    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w" event={"ID":"13f6ef0d-fbff-49c2-b187-d4340496ee8b","Type":"ContainerDied","Data":"d793c2237d65c37419b1d637f39a881b7ba53dbb7ddc89c82d94a62d448439ec"}
Nov 02 18:25:21.902860 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:21.902838    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"
Nov 02 18:25:22.018035 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:22.018004    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/13f6ef0d-fbff-49c2-b187-d4340496ee8b-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f6ef0d-fbff-49c2-b187-d4340496ee8b-maas-api-xks-ca\") pod \"13f6ef0d-fbff-49c2-b187-d4340496ee8b\" (UID: \"13f6ef0d-fbff-49c2-b187-d4340496ee8b\") "
Nov 02 18:25:22.018191 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:22.018045    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/13f6ef0d-fbff-49c2-b187-d4340496ee8b-kube-api-access-6n4cr\" (UniqueName: \"kubernetes.io/projected/13f6ef0d-fbff-49c2-b187-d4340496ee8b-kube-api-access-6n4cr\") pod \"13f6ef0d-fbff-49c2-b187-d4340496ee8b\" (UID: \"13f6ef0d-fbff-49c2-b187-d4340496ee8b\") "
Nov 02 18:25:22.018191 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:22.018175    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/13f6ef0d-fbff-49c2-b187-d4340496ee8b-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f6ef0d-fbff-49c2-b187-d4340496ee8b-maas-api-openshift-ca\") pod \"13f6ef0d-fbff-49c2-b187-d4340496ee8b\" (UID: \"13f6ef0d-fbff-49c2-b187-d4340496ee8b\") "
Nov 02 18:25:22.018552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:22.018526    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/13f6ef0d-fbff-49c2-b187-d4340496ee8b-maas-api-openshift-ca" pod "13f6ef0d-fbff-49c2-b187-d4340496ee8b" (UID: "13f6ef0d-fbff-49c2-b187-d4340496ee8b"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:25:22.020300 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:22.020278    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/13f6ef0d-fbff-49c2-b187-d4340496ee8b-kube-api-access-6n4cr" pod "13f6ef0d-fbff-49c2-b187-d4340496ee8b" (UID: "13f6ef0d-fbff-49c2-b187-d4340496ee8b"). InnerVolumeSpecName "kube-api-access-6n4cr". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:25:22.020426 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:22.020384    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/13f6ef0d-fbff-49c2-b187-d4340496ee8b-maas-api-xks-ca" pod "13f6ef0d-fbff-49c2-b187-d4340496ee8b" (UID: "13f6ef0d-fbff-49c2-b187-d4340496ee8b"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:25:22.119868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:22.119832    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-6n4cr\" (UniqueName: \"kubernetes.io/projected/13f6ef0d-fbff-49c2-b187-d4340496ee8b-kube-api-access-6n4cr\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:25:22.119868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:22.119861    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/13f6ef0d-fbff-49c2-b187-d4340496ee8b-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:25:22.119868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:22.119871    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/13f6ef0d-fbff-49c2-b187-d4340496ee8b-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:25:22.777952 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:22.777914    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w" event={"ID":"13f6ef0d-fbff-49c2-b187-d4340496ee8b","Type":"ContainerDied","Data":"8fc51ef50010664cb138c9511f23b6e2cc3e440047c95345c4e24119495bd4b6"}
Nov 02 18:25:22.778132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:22.777967    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"
Nov 02 18:25:22.778132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:22.777974    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="8fc51ef50010664cb138c9511f23b6e2cc3e440047c95345c4e24119495bd4b6"
Nov 02 18:25:26.032587 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:26.032556    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"]
Nov 02 18:25:26.045740 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:26.045712    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-0b7e8edb-tl5xf"]
Nov 02 18:25:27.612383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:27.612353    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="824f80b5-91a8-481e-a1c7-187595cff9fa" path="/var/lib/kubelet/pods/824f80b5-91a8-481e-a1c7-187595cff9fa/volumes"
Nov 02 18:25:39.656697 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:39.656657    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:39.860263 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:39.860218    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:40.004179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.004146    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:40.116156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.116119    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"]
Nov 02 18:25:40.116522 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.116510    4456 cpu_manager.go:414] "containerMap: removing container" podUID="13f6ef0d-fbff-49c2-b187-d4340496ee8b" containerName="revoke-keys"
Nov 02 18:25:40.116573 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.116524    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="13f6ef0d-fbff-49c2-b187-d4340496ee8b" containerName="revoke-keys"
Nov 02 18:25:40.116614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.116588    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="13f6ef0d-fbff-49c2-b187-d4340496ee8b" containerName="revoke-keys"
Nov 02 18:25:40.119293 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.119269    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"
Nov 02 18:25:40.139499 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.139467    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"]
Nov 02 18:25:40.149836 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.149789    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:25:40.150030 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.149867    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:25:40.150030 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.149922    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:25:40.150179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.149943    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:25:40.222808 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.222776    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ff46e3-3e04-4708-ade1-9097b6793298-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-v6ccg\" (UID: \"91ff46e3-3e04-4708-ade1-9097b6793298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"
Nov 02 18:25:40.222808 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.222807    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-mx4vq\" (UniqueName: \"kubernetes.io/projected/91ff46e3-3e04-4708-ade1-9097b6793298-kube-api-access-mx4vq\") pod \"maas-api-revoke-sub-d2bc93bc-v6ccg\" (UID: \"91ff46e3-3e04-4708-ade1-9097b6793298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"
Nov 02 18:25:40.223010 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.222976    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ff46e3-3e04-4708-ade1-9097b6793298-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-v6ccg\" (UID: \"91ff46e3-3e04-4708-ade1-9097b6793298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"
Nov 02 18:25:40.324237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.324154    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ff46e3-3e04-4708-ade1-9097b6793298-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-v6ccg\" (UID: \"91ff46e3-3e04-4708-ade1-9097b6793298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"
Nov 02 18:25:40.324237 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.324189    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-mx4vq\" (UniqueName: \"kubernetes.io/projected/91ff46e3-3e04-4708-ade1-9097b6793298-kube-api-access-mx4vq\") pod \"maas-api-revoke-sub-d2bc93bc-v6ccg\" (UID: \"91ff46e3-3e04-4708-ade1-9097b6793298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"
Nov 02 18:25:40.324405 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.324257    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ff46e3-3e04-4708-ade1-9097b6793298-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-v6ccg\" (UID: \"91ff46e3-3e04-4708-ade1-9097b6793298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"
Nov 02 18:25:40.325084 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.325061    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ff46e3-3e04-4708-ade1-9097b6793298-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-d2bc93bc-v6ccg\" (UID: \"91ff46e3-3e04-4708-ade1-9097b6793298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"
Nov 02 18:25:40.326770 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.326752    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ff46e3-3e04-4708-ade1-9097b6793298-maas-api-xks-ca\") pod \"maas-api-revoke-sub-d2bc93bc-v6ccg\" (UID: \"91ff46e3-3e04-4708-ade1-9097b6793298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"
Nov 02 18:25:40.349149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.349126    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-mx4vq\" (UniqueName: \"kubernetes.io/projected/91ff46e3-3e04-4708-ade1-9097b6793298-kube-api-access-mx4vq\") pod \"maas-api-revoke-sub-d2bc93bc-v6ccg\" (UID: \"91ff46e3-3e04-4708-ade1-9097b6793298\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"
Nov 02 18:25:40.431908 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.431877    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"
Nov 02 18:25:40.729229 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.729200    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"]
Nov 02 18:25:40.730268 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:25:40.730243    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod91ff46e3_3e04_4708_ade1_9097b6793298.slice/crio-b788d12ce65e450479541a60a3bc19f1d2300a56fc0cb2f6a8504fdb5b7aa862 WatchSource:0}: Error finding container b788d12ce65e450479541a60a3bc19f1d2300a56fc0cb2f6a8504fdb5b7aa862: Status 404 returned error can't find the container with id b788d12ce65e450479541a60a3bc19f1d2300a56fc0cb2f6a8504fdb5b7aa862
Nov 02 18:25:40.903844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:40.903801    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg" event={"ID":"91ff46e3-3e04-4708-ade1-9097b6793298","Type":"ContainerStarted","Data":"b788d12ce65e450479541a60a3bc19f1d2300a56fc0cb2f6a8504fdb5b7aa862"}
Nov 02 18:25:41.910610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:41.910515    4456 generic.go:356] "Generic (PLEG): container finished" podID="91ff46e3-3e04-4708-ade1-9097b6793298" containerID="011a1cee76f493227a0d5cb4603055f4bcc209fc89702118841db690ca2067fc" exitCode=0
Nov 02 18:25:41.910610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:41.910568    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg" event={"ID":"91ff46e3-3e04-4708-ade1-9097b6793298","Type":"ContainerDied","Data":"011a1cee76f493227a0d5cb4603055f4bcc209fc89702118841db690ca2067fc"}
Nov 02 18:25:43.055118 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:43.055090    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"
Nov 02 18:25:43.149088 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:43.149053    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/91ff46e3-3e04-4708-ade1-9097b6793298-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ff46e3-3e04-4708-ade1-9097b6793298-maas-api-openshift-ca\") pod \"91ff46e3-3e04-4708-ade1-9097b6793298\" (UID: \"91ff46e3-3e04-4708-ade1-9097b6793298\") "
Nov 02 18:25:43.149261 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:43.149187    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/91ff46e3-3e04-4708-ade1-9097b6793298-kube-api-access-mx4vq\" (UniqueName: \"kubernetes.io/projected/91ff46e3-3e04-4708-ade1-9097b6793298-kube-api-access-mx4vq\") pod \"91ff46e3-3e04-4708-ade1-9097b6793298\" (UID: \"91ff46e3-3e04-4708-ade1-9097b6793298\") "
Nov 02 18:25:43.149261 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:43.149249    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/91ff46e3-3e04-4708-ade1-9097b6793298-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ff46e3-3e04-4708-ade1-9097b6793298-maas-api-xks-ca\") pod \"91ff46e3-3e04-4708-ade1-9097b6793298\" (UID: \"91ff46e3-3e04-4708-ade1-9097b6793298\") "
Nov 02 18:25:43.149484 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:43.149461    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/91ff46e3-3e04-4708-ade1-9097b6793298-maas-api-openshift-ca" pod "91ff46e3-3e04-4708-ade1-9097b6793298" (UID: "91ff46e3-3e04-4708-ade1-9097b6793298"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:25:43.149593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:43.149578    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/91ff46e3-3e04-4708-ade1-9097b6793298-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:25:43.151501 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:43.151473    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/91ff46e3-3e04-4708-ade1-9097b6793298-maas-api-xks-ca" pod "91ff46e3-3e04-4708-ade1-9097b6793298" (UID: "91ff46e3-3e04-4708-ade1-9097b6793298"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:25:43.151610 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:43.151552    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/91ff46e3-3e04-4708-ade1-9097b6793298-kube-api-access-mx4vq" pod "91ff46e3-3e04-4708-ade1-9097b6793298" (UID: "91ff46e3-3e04-4708-ade1-9097b6793298"). InnerVolumeSpecName "kube-api-access-mx4vq". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:25:43.250397 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:43.250368    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-mx4vq\" (UniqueName: \"kubernetes.io/projected/91ff46e3-3e04-4708-ade1-9097b6793298-kube-api-access-mx4vq\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:25:43.250397 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:43.250393    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/91ff46e3-3e04-4708-ade1-9097b6793298-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:25:43.925455 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:43.925370    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"
Nov 02 18:25:43.925455 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:43.925404    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg" event={"ID":"91ff46e3-3e04-4708-ade1-9097b6793298","Type":"ContainerDied","Data":"b788d12ce65e450479541a60a3bc19f1d2300a56fc0cb2f6a8504fdb5b7aa862"}
Nov 02 18:25:43.925455 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:43.925451    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="b788d12ce65e450479541a60a3bc19f1d2300a56fc0cb2f6a8504fdb5b7aa862"
Nov 02 18:25:46.164464 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:25:46.164428    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:25:52.037232 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.037169    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:52.446846 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.446790    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"]
Nov 02 18:25:52.447487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.447464    4456 cpu_manager.go:414] "containerMap: removing container" podUID="91ff46e3-3e04-4708-ade1-9097b6793298" containerName="revoke-keys"
Nov 02 18:25:52.447487 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.447491    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="91ff46e3-3e04-4708-ade1-9097b6793298" containerName="revoke-keys"
Nov 02 18:25:52.447737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.447712    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="91ff46e3-3e04-4708-ade1-9097b6793298" containerName="revoke-keys"
Nov 02 18:25:52.451542 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.451517    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.467553 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.467533    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-worker-single-main-dbe6df-openshift-default-dockercfg-mqv27\""
Nov 02 18:25:52.483311 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.483281    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"]
Nov 02 18:25:52.537331 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.537300    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-workload-socket\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.537497 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.537352    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-podinfo\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.537548 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.537491    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae960905-dbfc-49d1-9013-936e9fe96c01-istiod-ca-cert\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.537592 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.537566    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-credential-socket\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.537631 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.537604    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-token\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.537707 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.537686    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-data\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.537753 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.537744    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-workload-certs\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.537807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.537794    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-envoy\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.537877 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.537867    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-4pflj\" (UniqueName: \"kubernetes.io/projected/ae960905-dbfc-49d1-9013-936e9fe96c01-kube-api-access-4pflj\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.537964 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.537946    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:52.638705 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.638659    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-envoy\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.638899 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.638755    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-4pflj\" (UniqueName: \"kubernetes.io/projected/ae960905-dbfc-49d1-9013-936e9fe96c01-kube-api-access-4pflj\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.638899 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.638847    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-workload-socket\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.638899 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.638886    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-podinfo\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.639077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.638928    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae960905-dbfc-49d1-9013-936e9fe96c01-istiod-ca-cert\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.639077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.638990    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-credential-socket\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.639077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.639036    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-token\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.639077 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.639066    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-data\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.639278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.639097    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-workload-certs\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.639278 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.639245    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-workload-socket\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.639447 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.639424    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-workload-certs\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.639513 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.639490    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-credential-socket\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.639711 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.639687    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-data\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.640224 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.640143    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae960905-dbfc-49d1-9013-936e9fe96c01-istiod-ca-cert\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.643944 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.642040    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-envoy\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.643944 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.642383    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-podinfo\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.668884 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.668840    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-token\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.669310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.669289    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-4pflj\" (UniqueName: \"kubernetes.io/projected/ae960905-dbfc-49d1-9013-936e9fe96c01-kube-api-access-4pflj\") pod \"e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") " pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:52.683706 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.683666    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:52.767708 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:52.767627    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:53.080099 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:53.080069    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"]
Nov 02 18:25:53.081234 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:25:53.081207    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podae960905_dbfc_49d1_9013_936e9fe96c01.slice/crio-a82bf1e89f23091038aacde9a6379f6e896d50ac395ca84f0e0f1e6b9bc3b5ae WatchSource:0}: Error finding container a82bf1e89f23091038aacde9a6379f6e896d50ac395ca84f0e0f1e6b9bc3b5ae: Status 404 returned error can't find the container with id a82bf1e89f23091038aacde9a6379f6e896d50ac395ca84f0e0f1e6b9bc3b5ae
Nov 02 18:25:53.083806 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:53.083776    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:25:53.083904 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:53.083862    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:25:53.083904 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:53.083894    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:25:53.981280 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:53.981238    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj" event={"ID":"ae960905-dbfc-49d1-9013-936e9fe96c01","Type":"ContainerStarted","Data":"a5fa4e52ba6314243ed9cd7ea2bf0c4da3c0462ec07adb1669aa3e57a9aa8f9f"}
Nov 02 18:25:53.981458 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:53.981303    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj" event={"ID":"ae960905-dbfc-49d1-9013-936e9fe96c01","Type":"ContainerStarted","Data":"a82bf1e89f23091038aacde9a6379f6e896d50ac395ca84f0e0f1e6b9bc3b5ae"}
Nov 02 18:25:54.042924 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:54.042861    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj" podStartSLOduration=2.042844912 podStartE2EDuration="2.042844912s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:52 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:25:54.041826129 +0000 UTC m=+3370.532889297" watchObservedRunningTime="2026-11-02 18:25:54.042844912 +0000 UTC m=+3370.533908072"
Nov 02 18:25:54.768772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:54.768733    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:54.775152 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:54.775118    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:54.987270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:54.987233    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:54.988308 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:54.988291    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:25:56.497092 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:56.497059    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:56.668891 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:56.668854    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:57.997733 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:57.997703    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"]
Nov 02 18:25:58.001354 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.001335    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:25:58.019690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.019638    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-worker-single-main-dbe6df-serving-cert\""
Nov 02 18:25:58.019938 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.019717    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 18:25:58.019938 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.019756    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:25:58.033295 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.033267    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gml5d\""
Nov 02 18:25:58.033295 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.033268    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:25:58.036105 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.036081    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"]
Nov 02 18:25:58.096133 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.096107    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:25:58.096267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.096151    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/748f59de-7f01-446a-8315-b4516fc7025d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:25:58.096267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.096179    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:25:58.096356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.096337    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-lz65p\" (UniqueName: \"kubernetes.io/projected/748f59de-7f01-446a-8315-b4516fc7025d-kube-api-access-lz65p\") pod \"maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:25:58.178022 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.177987    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:58.197334 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.197304    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:25:58.197500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.197347    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/748f59de-7f01-446a-8315-b4516fc7025d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:25:58.197500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.197373    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:25:58.197500 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:25:58.197470    4456 secret.go:189] Couldn't get secret odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-serving-cert: secret "maas-api-e2e-worker-single-main-dbe6df-serving-cert" not found
Nov 02 18:25:58.197500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.197487    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-lz65p\" (UniqueName: \"kubernetes.io/projected/748f59de-7f01-446a-8315-b4516fc7025d-kube-api-access-lz65p\") pod \"maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:25:58.197697 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:25:58.197541    4456 nestedpendingoperations.go:348] Operation for "{volumeName:kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-maas-api-tls podName:748f59de-7f01-446a-8315-b4516fc7025d nodeName:}" failed. No retries permitted until 2026-11-02 18:25:58.697520519 +0000 UTC m=+3375.188583658 (durationBeforeRetry 500ms). Error: MountVolume.SetUp failed for volume "maas-api-tls" (UniqueName: "kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-maas-api-tls") pod "maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8" (UID: "748f59de-7f01-446a-8315-b4516fc7025d") : secret "maas-api-e2e-worker-single-main-dbe6df-serving-cert" not found
Nov 02 18:25:58.198072 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.198055    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/748f59de-7f01-446a-8315-b4516fc7025d-openshift-service-ca\") pod \"maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:25:58.200166 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.200133    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-metrics-certs\") pod \"maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:25:58.224185 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.224163    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-lz65p\" (UniqueName: \"kubernetes.io/projected/748f59de-7f01-446a-8315-b4516fc7025d-kube-api-access-lz65p\") pod \"maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:25:58.316875 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.316762    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:58.455362 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.455332    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"]
Nov 02 18:25:58.459110 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.459082    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"
Nov 02 18:25:58.464288 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.464259    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:25:58.475926 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.475887    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-worker-single-main-dbe6df\""
Nov 02 18:25:58.476145 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.475975    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-worker-single-m-3b72f8da-dockercfg-vltdq\""
Nov 02 18:25:58.479844 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.479800    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"]
Nov 02 18:25:58.500257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.500237    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-l5tls\" (UniqueName: \"kubernetes.io/projected/8a3cbd30-aabd-446b-994c-b6ae1b581484-kube-api-access-l5tls\") pod \"payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84\" (UID: \"8a3cbd30-aabd-446b-994c-b6ae1b581484\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"
Nov 02 18:25:58.500353 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.500325    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3cbd30-aabd-446b-994c-b6ae1b581484-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84\" (UID: \"8a3cbd30-aabd-446b-994c-b6ae1b581484\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"
Nov 02 18:25:58.601721 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.601639    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-l5tls\" (UniqueName: \"kubernetes.io/projected/8a3cbd30-aabd-446b-994c-b6ae1b581484-kube-api-access-l5tls\") pod \"payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84\" (UID: \"8a3cbd30-aabd-446b-994c-b6ae1b581484\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"
Nov 02 18:25:58.601858 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.601733    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3cbd30-aabd-446b-994c-b6ae1b581484-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84\" (UID: \"8a3cbd30-aabd-446b-994c-b6ae1b581484\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"
Nov 02 18:25:58.602429 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.602410    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3cbd30-aabd-446b-994c-b6ae1b581484-plugins-config-volume\") pod \"payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84\" (UID: \"8a3cbd30-aabd-446b-994c-b6ae1b581484\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"
Nov 02 18:25:58.626617 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.626590    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-l5tls\" (UniqueName: \"kubernetes.io/projected/8a3cbd30-aabd-446b-994c-b6ae1b581484-kube-api-access-l5tls\") pod \"payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84\" (UID: \"8a3cbd30-aabd-446b-994c-b6ae1b581484\") " pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"
Nov 02 18:25:58.702776 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.702740    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:25:58.705283 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.705264    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-maas-api-tls\") pod \"maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") " pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:25:58.773311 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.773274    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"
Nov 02 18:25:58.915754 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:58.915675    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:25:59.071259 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:59.071234    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"]
Nov 02 18:25:59.071601 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:25:59.071571    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8a3cbd30_aabd_446b_994c_b6ae1b581484.slice/crio-f67b6e3459f20f7c3e43852965e7c75e2f7a0e4bbbed0383458d4839bd683f06 WatchSource:0}: Error finding container f67b6e3459f20f7c3e43852965e7c75e2f7a0e4bbbed0383458d4839bd683f06: Status 404 returned error can't find the container with id f67b6e3459f20f7c3e43852965e7c75e2f7a0e4bbbed0383458d4839bd683f06
Nov 02 18:25:59.217413 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:25:59.217368    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod748f59de_7f01_446a_8315_b4516fc7025d.slice/crio-453a04e3e137021c492da0fc9d129d321657f79357a83fa55bff697c633f2833 WatchSource:0}: Error finding container 453a04e3e137021c492da0fc9d129d321657f79357a83fa55bff697c633f2833: Status 404 returned error can't find the container with id 453a04e3e137021c492da0fc9d129d321657f79357a83fa55bff697c633f2833
Nov 02 18:25:59.217709 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:59.217688    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"]
Nov 02 18:25:59.950523 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:25:59.950492    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:00.018697 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:00.018653    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8" event={"ID":"748f59de-7f01-446a-8315-b4516fc7025d","Type":"ContainerStarted","Data":"1bad47f8eba60d9ccc4ad8e819cb73036a24812e97f82ecf605d44a4164ea83a"}
Nov 02 18:26:00.018897 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:00.018739    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:26:00.018897 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:00.018769    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8" event={"ID":"748f59de-7f01-446a-8315-b4516fc7025d","Type":"ContainerStarted","Data":"453a04e3e137021c492da0fc9d129d321657f79357a83fa55bff697c633f2833"}
Nov 02 18:26:00.020448 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:00.020418    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84" event={"ID":"8a3cbd30-aabd-446b-994c-b6ae1b581484","Type":"ContainerStarted","Data":"bccc3110494c6b48dc8b08728e073909599abeae052e9b0bb7c3341396e1136b"}
Nov 02 18:26:00.020552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:00.020468    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"
Nov 02 18:26:00.020552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:00.020486    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84" event={"ID":"8a3cbd30-aabd-446b-994c-b6ae1b581484","Type":"ContainerStarted","Data":"f67b6e3459f20f7c3e43852965e7c75e2f7a0e4bbbed0383458d4839bd683f06"}
Nov 02 18:26:00.064575 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:00.064532    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8" podStartSLOduration=2.613976449 podStartE2EDuration="3.064519676s" totalImagesPullingTime="450.543227ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:57 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:00.06439217 +0000 UTC m=+3376.555455332" watchObservedRunningTime="2026-11-02 18:26:00.064519676 +0000 UTC m=+3376.555582837"
Nov 02 18:26:00.108135 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:00.108077    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84" podStartSLOduration=2.108060801 podStartE2EDuration="2.108060801s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:25:58 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:26:00.10670663 +0000 UTC m=+3376.597769793" watchObservedRunningTime="2026-11-02 18:26:00.108060801 +0000 UTC m=+3376.599123961"
Nov 02 18:26:01.773977 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:01.773933    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:03.652977 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:03.652945    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:05.449361 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:05.449331    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:06.032802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:06.032776    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:26:07.267324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:07.267292    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:09.164638 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:09.164594    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:10.887952 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:10.887869    4456 patch_prober.go:29] interesting pod/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": read tcp 10.134.0.2:41196->10.134.1.47:15021: read: connection reset by peer" start-of-body=
Nov 02 18:26:10.887952 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:10.887939    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj" podUID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": read tcp 10.134.0.2:41196->10.134.1.47:15021: read: connection reset by peer"
Nov 02 18:26:10.957694 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:10.957657    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:11.030627 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:11.030551    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"
Nov 02 18:26:11.083807 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:11.083765    4456 generic.go:356] "Generic (PLEG): container finished" podID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerID="a5fa4e52ba6314243ed9cd7ea2bf0c4da3c0462ec07adb1669aa3e57a9aa8f9f" exitCode=0
Nov 02 18:26:11.083993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:11.083883    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj" event={"ID":"ae960905-dbfc-49d1-9013-936e9fe96c01","Type":"ContainerDied","Data":"a5fa4e52ba6314243ed9cd7ea2bf0c4da3c0462ec07adb1669aa3e57a9aa8f9f"}
Nov 02 18:26:11.084332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:11.084308    4456 scope.go:122] "RemoveContainer" containerID="a5fa4e52ba6314243ed9cd7ea2bf0c4da3c0462ec07adb1669aa3e57a9aa8f9f"
Nov 02 18:26:11.085805 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:11.085771    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:26:11.085919 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:11.085892    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:26:11.085957 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:11.085940    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:26:11.347294 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:11.347263    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-driver/0.log"
Nov 02 18:26:11.377017 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:11.376980    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:11.403173 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:11.403094    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-driver/0.log"
Nov 02 18:26:11.999450 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:11.999414    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:12.109088 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:12.109054    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj" event={"ID":"ae960905-dbfc-49d1-9013-936e9fe96c01","Type":"ContainerStarted","Data":"80dad4e2cff4a4e428ef5dfd7f7f50d5c9df1a6b32cdd21f25b18f1a708d7926"}
Nov 02 18:26:12.155546 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:12.155518    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:12.315894 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:12.315798    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:12.767861 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:12.767798    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:26:12.768068 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:12.768007    4456 patch_prober.go:29] interesting pod/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body=
Nov 02 18:26:12.768145 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:12.768069    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj" podUID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused"
Nov 02 18:26:13.040158 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:13.040072    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:13.618190 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:13.618157    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:13.768053 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:13.768016    4456 patch_prober.go:29] interesting pod/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body=
Nov 02 18:26:13.768222 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:13.768089    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj" podUID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused"
Nov 02 18:26:14.124555 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:14.124516    4456 scope.go:122] "RemoveContainer" containerID="236134598a43af8d1d5f9fd29245771288fdaf2bfb9c3bec0dd1bab979b56ffb"
Nov 02 18:26:14.133729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:14.133704    4456 scope.go:122] "RemoveContainer" containerID="718dc6dfa59704bf6ed0ccfd19ff998fb4b685a22684857200a0cac92e794130"
Nov 02 18:26:14.142393 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:14.142363    4456 scope.go:122] "RemoveContainer" containerID="b5bd2f95da9c4d77655ae7512e2d4e78e3a21a7ddaf00fa2483d959637990940"
Nov 02 18:26:14.768842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:14.768791    4456 patch_prober.go:29] interesting pod/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body=
Nov 02 18:26:14.769032 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:14.768875    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj" podUID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused"
Nov 02 18:26:14.948712 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:14.948674    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:15.768523 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:15.768491    4456 patch_prober.go:29] interesting pod/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body=
Nov 02 18:26:15.768942 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:15.768554    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj" podUID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused"
Nov 02 18:26:16.168443 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:26:16.168345    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:26:16.768190 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:16.768157    4456 patch_prober.go:29] interesting pod/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused" start-of-body=
Nov 02 18:26:16.768380 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:16.768211    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj" podUID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.47:15021/healthz/ready\": dial tcp 10.134.1.47:15021: connect: connection refused"
Nov 02 18:26:16.852172 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:16.852124    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:17.771538 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:17.771506    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:26:17.771991 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:17.771972    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:26:17.772624 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:17.772606    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:26:18.660295 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:18.660261    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:20.553944 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:20.553910    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:22.361048 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:22.361011    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:24.170935 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:24.170886    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:25.961241 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:25.961198    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:26.031530 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:26.031500    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"]
Nov 02 18:26:26.044523 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:26.044495    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-e2fc9c5b-qqpfs"]
Nov 02 18:26:27.619831 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:27.619785    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d889d54c-fb96-4cf0-8d44-17485904481d" path="/var/lib/kubelet/pods/d889d54c-fb96-4cf0-8d44-17485904481d/volumes"
Nov 02 18:26:27.763917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:27.763880    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:29.559722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:29.559685    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:31.365642 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:31.365612    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:33.253877 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:33.253839    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:35.060677 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:35.060641    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:36.032098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:36.032064    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"]
Nov 02 18:26:36.043667 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:36.043643    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-5a29efb3-xg6sn"]
Nov 02 18:26:36.868714 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:36.868660    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:37.612250 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:37.612217    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="11dc527f-427b-4662-a56b-35ee92097ce2" path="/var/lib/kubelet/pods/11dc527f-427b-4662-a56b-35ee92097ce2/volumes"
Nov 02 18:26:38.752763 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:38.752724    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:39.813489 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:39.813453    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:40.655393 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:40.655356    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:40.796790 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:40.796759    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:42.541385 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:42.541352    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:44.454514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:44.454470    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:46.166377 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:26:46.166331    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:26:46.349759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:46.349725    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:48.155364 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:48.155327    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:49.943769 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:49.943734    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:51.744408 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:51.744369    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:53.575266 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:53.575229    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:54.896065 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:54.896029    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:55.424590 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:55.424554    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:57.439625 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:57.439583    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:59.254135 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:59.254104    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:26:59.930774 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:26:59.930732    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:01.051236 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:01.051197    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:02.866448 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:02.866416    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:04.775268 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:04.775230    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:05.628545 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:05.628513    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"]
Nov 02 18:27:05.796676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:05.796644    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:05.937282 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:05.937246    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:06.570346 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:06.570305    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:07.982614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:07.982578    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"]
Nov 02 18:27:07.983036 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:07.982833    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8" podUID="748f59de-7f01-446a-8315-b4516fc7025d" containerName="maas-api" containerID="cri-o://1bad47f8eba60d9ccc4ad8e819cb73036a24812e97f82ecf605d44a4164ea83a" gracePeriod=30
Nov 02 18:27:08.081447 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.081412    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:08.423554 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.423522    4456 generic.go:356] "Generic (PLEG): container finished" podID="748f59de-7f01-446a-8315-b4516fc7025d" containerID="1bad47f8eba60d9ccc4ad8e819cb73036a24812e97f82ecf605d44a4164ea83a" exitCode=0
Nov 02 18:27:08.423714 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.423593    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8" event={"ID":"748f59de-7f01-446a-8315-b4516fc7025d","Type":"ContainerDied","Data":"1bad47f8eba60d9ccc4ad8e819cb73036a24812e97f82ecf605d44a4164ea83a"}
Nov 02 18:27:08.463614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.463590    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:27:08.593997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.593918    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/748f59de-7f01-446a-8315-b4516fc7025d-kube-api-access-lz65p\" (UniqueName: \"kubernetes.io/projected/748f59de-7f01-446a-8315-b4516fc7025d-kube-api-access-lz65p\") pod \"748f59de-7f01-446a-8315-b4516fc7025d\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") "
Nov 02 18:27:08.594156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.594003    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/748f59de-7f01-446a-8315-b4516fc7025d-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/748f59de-7f01-446a-8315-b4516fc7025d-openshift-service-ca\") pod \"748f59de-7f01-446a-8315-b4516fc7025d\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") "
Nov 02 18:27:08.594156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.594063    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-metrics-certs\" (UniqueName: \"kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-metrics-certs\") pod \"748f59de-7f01-446a-8315-b4516fc7025d\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") "
Nov 02 18:27:08.594156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.594129    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-maas-api-tls\") pod \"748f59de-7f01-446a-8315-b4516fc7025d\" (UID: \"748f59de-7f01-446a-8315-b4516fc7025d\") "
Nov 02 18:27:08.594443 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.594414    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/748f59de-7f01-446a-8315-b4516fc7025d-openshift-service-ca" pod "748f59de-7f01-446a-8315-b4516fc7025d" (UID: "748f59de-7f01-446a-8315-b4516fc7025d"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:08.596517 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.596486    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-metrics-certs" pod "748f59de-7f01-446a-8315-b4516fc7025d" (UID: "748f59de-7f01-446a-8315-b4516fc7025d"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:27:08.596647 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.596532    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-maas-api-tls" pod "748f59de-7f01-446a-8315-b4516fc7025d" (UID: "748f59de-7f01-446a-8315-b4516fc7025d"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:27:08.596690 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.596663    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/748f59de-7f01-446a-8315-b4516fc7025d-kube-api-access-lz65p" pod "748f59de-7f01-446a-8315-b4516fc7025d" (UID: "748f59de-7f01-446a-8315-b4516fc7025d"). InnerVolumeSpecName "kube-api-access-lz65p". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:08.672666 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.672630    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:08.695268 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.695240    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-lz65p\" (UniqueName: \"kubernetes.io/projected/748f59de-7f01-446a-8315-b4516fc7025d-kube-api-access-lz65p\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:08.695268 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.695267    4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/748f59de-7f01-446a-8315-b4516fc7025d-openshift-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:08.695445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.695284    4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-metrics-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:08.695445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.695297    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/748f59de-7f01-446a-8315-b4516fc7025d-maas-api-tls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:08.910524 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:08.910430    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:09.321925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:09.321888    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:09.431599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:09.431570    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"
Nov 02 18:27:09.431789 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:09.431605    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8" event={"ID":"748f59de-7f01-446a-8315-b4516fc7025d","Type":"ContainerDied","Data":"453a04e3e137021c492da0fc9d129d321657f79357a83fa55bff697c633f2833"}
Nov 02 18:27:09.431789 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:09.431668    4456 scope.go:122] "RemoveContainer" containerID="1bad47f8eba60d9ccc4ad8e819cb73036a24812e97f82ecf605d44a4164ea83a"
Nov 02 18:27:09.488602 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:09.488569    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"]
Nov 02 18:27:09.503514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:09.503479    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-worker-single-main-dbe6df-6678dc5df7-85ww8"]
Nov 02 18:27:09.613307 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:09.613225    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="748f59de-7f01-446a-8315-b4516fc7025d" path="/var/lib/kubelet/pods/748f59de-7f01-446a-8315-b4516fc7025d/volumes"
Nov 02 18:27:10.629098 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:10.629016    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84" podUID="8a3cbd30-aabd-446b-994c-b6ae1b581484" containerName="payload-processing" containerID="cri-o://bccc3110494c6b48dc8b08728e073909599abeae052e9b0bb7c3341396e1136b" gracePeriod=25
Nov 02 18:27:11.028618 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:11.028574    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84" podUID="8a3cbd30-aabd-446b-994c-b6ae1b581484" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:27:14.171533 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:14.171501    4456 scope.go:122] "RemoveContainer" containerID="0157f8fcb991b9a78b3c20b43afd7d18dd2bd63ff3aa7a038270dd33066baa69"
Nov 02 18:27:14.180508 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:14.180488    4456 scope.go:122] "RemoveContainer" containerID="ac0ca6e8b0f60e96625dd9ef0f3cbbd1fce77906efac8a6598c12fddfd4095c9"
Nov 02 18:27:14.188393 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:14.188371    4456 scope.go:122] "RemoveContainer" containerID="ed7641405fbeccbad1389c47987a6ca34d7908619995bd2d6acd428fe2992a56"
Nov 02 18:27:15.303939 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:15.303899    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:15.443943 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:15.443900    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:15.589699 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:15.589611    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"]
Nov 02 18:27:15.589951 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:15.589922    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj" podUID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerName="istio-proxy" containerID="cri-o://80dad4e2cff4a4e428ef5dfd7f7f50d5c9df1a6b32cdd21f25b18f1a708d7926" gracePeriod=30
Nov 02 18:27:15.652994 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:15.652960    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:16.164978 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:27:16.164936    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:27:17.774621 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:17.774588    4456 patch_prober.go:29] interesting pod/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="HTTP probe failed with statuscode: 503" start-of-body=
Nov 02 18:27:17.775112 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:17.774661    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj" podUID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerName="istio-proxy" probeResult="failure" output="HTTP probe failed with statuscode: 503"
Nov 02 18:27:19.922304 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:19.922253    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:20.126105 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.126066    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:20.261083 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.261051    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:20.855338 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.855279    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:27:20.920594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.918592    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-envoy\") pod \"ae960905-dbfc-49d1-9013-936e9fe96c01\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") "
Nov 02 18:27:20.920594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.918729    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-token\" (UniqueName: \"kubernetes.io/projected/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-token\") pod \"ae960905-dbfc-49d1-9013-936e9fe96c01\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") "
Nov 02 18:27:20.920594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.918779    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/ae960905-dbfc-49d1-9013-936e9fe96c01-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae960905-dbfc-49d1-9013-936e9fe96c01-istiod-ca-cert\") pod \"ae960905-dbfc-49d1-9013-936e9fe96c01\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") "
Nov 02 18:27:20.920594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.918831    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-workload-socket\") pod \"ae960905-dbfc-49d1-9013-936e9fe96c01\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") "
Nov 02 18:27:20.920594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.918874    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-data\") pod \"ae960905-dbfc-49d1-9013-936e9fe96c01\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") "
Nov 02 18:27:20.921456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.918903    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/ae960905-dbfc-49d1-9013-936e9fe96c01-kube-api-access-4pflj\" (UniqueName: \"kubernetes.io/projected/ae960905-dbfc-49d1-9013-936e9fe96c01-kube-api-access-4pflj\") pod \"ae960905-dbfc-49d1-9013-936e9fe96c01\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") "
Nov 02 18:27:20.921456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.918948    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-podinfo\") pod \"ae960905-dbfc-49d1-9013-936e9fe96c01\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") "
Nov 02 18:27:20.921456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.918996    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-workload-certs\") pod \"ae960905-dbfc-49d1-9013-936e9fe96c01\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") "
Nov 02 18:27:20.921456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.919076    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-credential-socket\") pod \"ae960905-dbfc-49d1-9013-936e9fe96c01\" (UID: \"ae960905-dbfc-49d1-9013-936e9fe96c01\") "
Nov 02 18:27:20.921456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.919865    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-workload-socket" pod "ae960905-dbfc-49d1-9013-936e9fe96c01" (UID: "ae960905-dbfc-49d1-9013-936e9fe96c01"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:20.921731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.920082    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-workload-certs" pod "ae960905-dbfc-49d1-9013-936e9fe96c01" (UID: "ae960905-dbfc-49d1-9013-936e9fe96c01"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:20.921731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.920156    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-credential-socket" pod "ae960905-dbfc-49d1-9013-936e9fe96c01" (UID: "ae960905-dbfc-49d1-9013-936e9fe96c01"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:20.921731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.920207    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/ae960905-dbfc-49d1-9013-936e9fe96c01-istiod-ca-cert" pod "ae960905-dbfc-49d1-9013-936e9fe96c01" (UID: "ae960905-dbfc-49d1-9013-936e9fe96c01"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:20.921731 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.920441    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-data" pod "ae960905-dbfc-49d1-9013-936e9fe96c01" (UID: "ae960905-dbfc-49d1-9013-936e9fe96c01"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:20.923129 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.923100    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-token" pod "ae960905-dbfc-49d1-9013-936e9fe96c01" (UID: "ae960905-dbfc-49d1-9013-936e9fe96c01"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:20.923574 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.923191    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-podinfo" pod "ae960905-dbfc-49d1-9013-936e9fe96c01" (UID: "ae960905-dbfc-49d1-9013-936e9fe96c01"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:27:20.924360 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.924336    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/ae960905-dbfc-49d1-9013-936e9fe96c01-kube-api-access-4pflj" pod "ae960905-dbfc-49d1-9013-936e9fe96c01" (UID: "ae960905-dbfc-49d1-9013-936e9fe96c01"). InnerVolumeSpecName "kube-api-access-4pflj". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:20.925009 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:20.924987    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-envoy" pod "ae960905-dbfc-49d1-9013-936e9fe96c01" (UID: "ae960905-dbfc-49d1-9013-936e9fe96c01"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:27:21.019970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.019942    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:21.019970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.019968    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-4pflj\" (UniqueName: \"kubernetes.io/projected/ae960905-dbfc-49d1-9013-936e9fe96c01-kube-api-access-4pflj\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:21.019970 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.019979    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:21.020200 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.019988    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:21.020200 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.019996    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:21.020200 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.020005    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:21.020200 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.020013    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/ae960905-dbfc-49d1-9013-936e9fe96c01-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:21.020200 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.020020    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/ae960905-dbfc-49d1-9013-936e9fe96c01-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:21.020200 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.020027    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/ae960905-dbfc-49d1-9013-936e9fe96c01-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:21.028805 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.028771    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84" podUID="8a3cbd30-aabd-446b-994c-b6ae1b581484" containerName="payload-processing" probeResult="failure" output="service unhealthy (responded with \"NOT_SERVING\")"
Nov 02 18:27:21.497180 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.497147    4456 generic.go:356] "Generic (PLEG): container finished" podID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerID="80dad4e2cff4a4e428ef5dfd7f7f50d5c9df1a6b32cdd21f25b18f1a708d7926" exitCode=0
Nov 02 18:27:21.497341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.497207    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"
Nov 02 18:27:21.497341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.497227    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj" event={"ID":"ae960905-dbfc-49d1-9013-936e9fe96c01","Type":"ContainerDied","Data":"80dad4e2cff4a4e428ef5dfd7f7f50d5c9df1a6b32cdd21f25b18f1a708d7926"}
Nov 02 18:27:21.497341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.497279    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj" event={"ID":"ae960905-dbfc-49d1-9013-936e9fe96c01","Type":"ContainerDied","Data":"a82bf1e89f23091038aacde9a6379f6e896d50ac395ca84f0e0f1e6b9bc3b5ae"}
Nov 02 18:27:21.497341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.497301    4456 scope.go:122] "RemoveContainer" containerID="80dad4e2cff4a4e428ef5dfd7f7f50d5c9df1a6b32cdd21f25b18f1a708d7926"
Nov 02 18:27:21.507595 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.507578    4456 scope.go:122] "RemoveContainer" containerID="a5fa4e52ba6314243ed9cd7ea2bf0c4da3c0462ec07adb1669aa3e57a9aa8f9f"
Nov 02 18:27:21.515917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.515900    4456 scope.go:122] "RemoveContainer" containerID="80dad4e2cff4a4e428ef5dfd7f7f50d5c9df1a6b32cdd21f25b18f1a708d7926"
Nov 02 18:27:21.516194 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:27:21.516173    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"80dad4e2cff4a4e428ef5dfd7f7f50d5c9df1a6b32cdd21f25b18f1a708d7926\": container with ID starting with 80dad4e2cff4a4e428ef5dfd7f7f50d5c9df1a6b32cdd21f25b18f1a708d7926 not found: ID does not exist" containerID="80dad4e2cff4a4e428ef5dfd7f7f50d5c9df1a6b32cdd21f25b18f1a708d7926"
Nov 02 18:27:21.516270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.516205    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"80dad4e2cff4a4e428ef5dfd7f7f50d5c9df1a6b32cdd21f25b18f1a708d7926"} err="failed to get container status \"80dad4e2cff4a4e428ef5dfd7f7f50d5c9df1a6b32cdd21f25b18f1a708d7926\": rpc error: code = NotFound desc = could not find container \"80dad4e2cff4a4e428ef5dfd7f7f50d5c9df1a6b32cdd21f25b18f1a708d7926\": container with ID starting with 80dad4e2cff4a4e428ef5dfd7f7f50d5c9df1a6b32cdd21f25b18f1a708d7926 not found: ID does not exist"
Nov 02 18:27:21.516270 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.516229    4456 scope.go:122] "RemoveContainer" containerID="a5fa4e52ba6314243ed9cd7ea2bf0c4da3c0462ec07adb1669aa3e57a9aa8f9f"
Nov 02 18:27:21.516484 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:27:21.516466    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"a5fa4e52ba6314243ed9cd7ea2bf0c4da3c0462ec07adb1669aa3e57a9aa8f9f\": container with ID starting with a5fa4e52ba6314243ed9cd7ea2bf0c4da3c0462ec07adb1669aa3e57a9aa8f9f not found: ID does not exist" containerID="a5fa4e52ba6314243ed9cd7ea2bf0c4da3c0462ec07adb1669aa3e57a9aa8f9f"
Nov 02 18:27:21.516522 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.516491    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"a5fa4e52ba6314243ed9cd7ea2bf0c4da3c0462ec07adb1669aa3e57a9aa8f9f"} err="failed to get container status \"a5fa4e52ba6314243ed9cd7ea2bf0c4da3c0462ec07adb1669aa3e57a9aa8f9f\": rpc error: code = NotFound desc = could not find container \"a5fa4e52ba6314243ed9cd7ea2bf0c4da3c0462ec07adb1669aa3e57a9aa8f9f\": container with ID starting with a5fa4e52ba6314243ed9cd7ea2bf0c4da3c0462ec07adb1669aa3e57a9aa8f9f not found: ID does not exist"
Nov 02 18:27:21.566890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.566858    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"]
Nov 02 18:27:21.569905 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.569873    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-worker-single-main-dbe6df-openshift-default-7fbff7b689fh9dj"]
Nov 02 18:27:21.613523 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:21.613492    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="ae960905-dbfc-49d1-9013-936e9fe96c01" path="/var/lib/kubelet/pods/ae960905-dbfc-49d1-9013-936e9fe96c01/volumes"
Nov 02 18:27:30.874352 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:30.874326    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"
Nov 02 18:27:30.929877 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:30.929782    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8a3cbd30-aabd-446b-994c-b6ae1b581484-kube-api-access-l5tls\" (UniqueName: \"kubernetes.io/projected/8a3cbd30-aabd-446b-994c-b6ae1b581484-kube-api-access-l5tls\") pod \"8a3cbd30-aabd-446b-994c-b6ae1b581484\" (UID: \"8a3cbd30-aabd-446b-994c-b6ae1b581484\") "
Nov 02 18:27:30.930037 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:30.929996    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8a3cbd30-aabd-446b-994c-b6ae1b581484-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3cbd30-aabd-446b-994c-b6ae1b581484-plugins-config-volume\") pod \"8a3cbd30-aabd-446b-994c-b6ae1b581484\" (UID: \"8a3cbd30-aabd-446b-994c-b6ae1b581484\") "
Nov 02 18:27:30.930593 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:30.930562    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8a3cbd30-aabd-446b-994c-b6ae1b581484-plugins-config-volume" pod "8a3cbd30-aabd-446b-994c-b6ae1b581484" (UID: "8a3cbd30-aabd-446b-994c-b6ae1b581484"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:30.932827 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:30.932787    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8a3cbd30-aabd-446b-994c-b6ae1b581484-kube-api-access-l5tls" pod "8a3cbd30-aabd-446b-994c-b6ae1b581484" (UID: "8a3cbd30-aabd-446b-994c-b6ae1b581484"). InnerVolumeSpecName "kube-api-access-l5tls". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:31.031752 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:31.031720    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-l5tls\" (UniqueName: \"kubernetes.io/projected/8a3cbd30-aabd-446b-994c-b6ae1b581484-kube-api-access-l5tls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:31.031752 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:31.031750    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/8a3cbd30-aabd-446b-994c-b6ae1b581484-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:31.573457 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:31.573421    4456 generic.go:356] "Generic (PLEG): container finished" podID="8a3cbd30-aabd-446b-994c-b6ae1b581484" containerID="bccc3110494c6b48dc8b08728e073909599abeae052e9b0bb7c3341396e1136b" exitCode=0
Nov 02 18:27:31.573670 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:31.573503    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"
Nov 02 18:27:31.573670 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:31.573538    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84" event={"ID":"8a3cbd30-aabd-446b-994c-b6ae1b581484","Type":"ContainerDied","Data":"bccc3110494c6b48dc8b08728e073909599abeae052e9b0bb7c3341396e1136b"}
Nov 02 18:27:31.573670 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:31.573601    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84" event={"ID":"8a3cbd30-aabd-446b-994c-b6ae1b581484","Type":"ContainerDied","Data":"f67b6e3459f20f7c3e43852965e7c75e2f7a0e4bbbed0383458d4839bd683f06"}
Nov 02 18:27:31.573670 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:31.573625    4456 scope.go:122] "RemoveContainer" containerID="bccc3110494c6b48dc8b08728e073909599abeae052e9b0bb7c3341396e1136b"
Nov 02 18:27:31.582798 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:31.582782    4456 scope.go:122] "RemoveContainer" containerID="bccc3110494c6b48dc8b08728e073909599abeae052e9b0bb7c3341396e1136b"
Nov 02 18:27:31.583125 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:27:31.583103    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"bccc3110494c6b48dc8b08728e073909599abeae052e9b0bb7c3341396e1136b\": container with ID starting with bccc3110494c6b48dc8b08728e073909599abeae052e9b0bb7c3341396e1136b not found: ID does not exist" containerID="bccc3110494c6b48dc8b08728e073909599abeae052e9b0bb7c3341396e1136b"
Nov 02 18:27:31.583199 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:31.583132    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"bccc3110494c6b48dc8b08728e073909599abeae052e9b0bb7c3341396e1136b"} err="failed to get container status \"bccc3110494c6b48dc8b08728e073909599abeae052e9b0bb7c3341396e1136b\": rpc error: code = NotFound desc = could not find container \"bccc3110494c6b48dc8b08728e073909599abeae052e9b0bb7c3341396e1136b\": container with ID starting with bccc3110494c6b48dc8b08728e073909599abeae052e9b0bb7c3341396e1136b not found: ID does not exist"
Nov 02 18:27:31.632478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:31.632453    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"]
Nov 02 18:27:31.648835 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:31.648790    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-worker-single-main-dbe6df-76899455bshh84"]
Nov 02 18:27:33.612621 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:33.612590    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8a3cbd30-aabd-446b-994c-b6ae1b581484" path="/var/lib/kubelet/pods/8a3cbd30-aabd-446b-994c-b6ae1b581484/volumes"
Nov 02 18:27:36.001797 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.001763    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:36.549423 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.549383    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"]
Nov 02 18:27:36.550011 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.549991    4456 cpu_manager.go:414] "containerMap: removing container" podUID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerName="istio-proxy"
Nov 02 18:27:36.550071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.550016    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerName="istio-proxy"
Nov 02 18:27:36.550071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.550032    4456 cpu_manager.go:414] "containerMap: removing container" podUID="748f59de-7f01-446a-8315-b4516fc7025d" containerName="maas-api"
Nov 02 18:27:36.550071 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.550041    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="748f59de-7f01-446a-8315-b4516fc7025d" containerName="maas-api"
Nov 02 18:27:36.550161 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.550075    4456 cpu_manager.go:414] "containerMap: removing container" podUID="8a3cbd30-aabd-446b-994c-b6ae1b581484" containerName="payload-processing"
Nov 02 18:27:36.550161 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.550083    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8a3cbd30-aabd-446b-994c-b6ae1b581484" containerName="payload-processing"
Nov 02 18:27:36.550225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.550216    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="748f59de-7f01-446a-8315-b4516fc7025d" containerName="maas-api"
Nov 02 18:27:36.550256 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.550234    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8a3cbd30-aabd-446b-994c-b6ae1b581484" containerName="payload-processing"
Nov 02 18:27:36.550256 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.550250    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerName="istio-proxy"
Nov 02 18:27:36.550325 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.550262    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerName="istio-proxy"
Nov 02 18:27:36.555167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.555142    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.571667 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.571645    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"]
Nov 02 18:27:36.582546 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.582524    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"llm\"/\"e2e-same-id-b-d04a7d07-kserve-self-signed-certs\""
Nov 02 18:27:36.643648 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.643603    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:36.691175 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.691153    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-model-cache\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.691306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.691186    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-dshm\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.691306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.691232    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-home\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.691378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.691310    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-tmp-dir\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.691378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.691348    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2mtd8\" (UniqueName: \"kubernetes.io/projected/787de320-4dca-4811-affc-37617eba4965-kube-api-access-2mtd8\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.691378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.691373    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787de320-4dca-4811-affc-37617eba4965-tls-certs\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.792518 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.792480    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-home\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.792737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.792537    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-tmp-dir\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.792737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.792574    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2mtd8\" (UniqueName: \"kubernetes.io/projected/787de320-4dca-4811-affc-37617eba4965-kube-api-access-2mtd8\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.792737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.792608    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787de320-4dca-4811-affc-37617eba4965-tls-certs\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.792737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.792699    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-model-cache\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.792737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.792737    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-dshm\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.793046 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.792994    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-home\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.793100 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.793046    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-tmp-dir\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.793100 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.793079    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-model-cache\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.795214 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.795186    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-dshm\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.795500 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.795480    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787de320-4dca-4811-affc-37617eba4965-tls-certs\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.819099 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.819049    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2mtd8\" (UniqueName: \"kubernetes.io/projected/787de320-4dca-4811-affc-37617eba4965-kube-api-access-2mtd8\") pod \"e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") " pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:36.870097 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:36.870068    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:37.167175 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:37.167146    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"]
Nov 02 18:27:37.167788 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:27:37.167761    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-besteffort.slice/kubepods-besteffort-pod787de320_4dca_4811_affc_37617eba4965.slice/crio-dcc83ea6c51382b2629afc76858beff0472256f98244b458e6f2c001150fc07d WatchSource:0}: Error finding container dcc83ea6c51382b2629afc76858beff0472256f98244b458e6f2c001150fc07d: Status 404 returned error can't find the container with id dcc83ea6c51382b2629afc76858beff0472256f98244b458e6f2c001150fc07d
Nov 02 18:27:37.481041 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:37.480982    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:37.611580 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:37.611545    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw" event={"ID":"787de320-4dca-4811-affc-37617eba4965","Type":"ContainerStarted","Data":"8e6acff93f0e12722674839db84d0cf750d39430ef33731fc85190ca00990fc5"}
Nov 02 18:27:37.611741 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:37.611590    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw" event={"ID":"787de320-4dca-4811-affc-37617eba4965","Type":"ContainerStarted","Data":"dcc83ea6c51382b2629afc76858beff0472256f98244b458e6f2c001150fc07d"}
Nov 02 18:27:37.649419 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:37.649374    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw" podStartSLOduration=1.649359925 podStartE2EDuration="1.649359925s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:37.649090872 +0000 UTC m=+3474.140154028" watchObservedRunningTime="2026-11-02 18:27:37.649359925 +0000 UTC m=+3474.140423088"
Nov 02 18:27:37.870840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:37.870734    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:39.886675 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:39.886628    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:42.751383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:42.750958    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lrssd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:27:42.751383 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:42.751029    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd" podUID="9d114747-c309-445d-8159-5e532bede175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:27:46.165258 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:27:46.165221    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:27:46.871006 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:46.870974    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:46.883634 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:46.883605    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:47.787322 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:47.787293    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:27:52.671032 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:52.670992    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:27:53.636102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.636065    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"]
Nov 02 18:27:53.636464 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.636451    4456 cpu_manager.go:414] "containerMap: removing container" podUID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerName="istio-proxy"
Nov 02 18:27:53.636536 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.636465    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="ae960905-dbfc-49d1-9013-936e9fe96c01" containerName="istio-proxy"
Nov 02 18:27:53.641897 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.640868    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"
Nov 02 18:27:53.660978 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.660955    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"]
Nov 02 18:27:53.669525 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.669501    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:27:53.669639 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.669548    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:27:53.669639 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.669584    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:27:53.670079 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.670064    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:27:53.762809 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.762768    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-2pgkd\" (UniqueName: \"kubernetes.io/projected/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-kube-api-access-2pgkd\") pod \"maas-api-revoke-sub-452b6d7b-p9mpc\" (UID: \"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"
Nov 02 18:27:53.763298 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.762920    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-p9mpc\" (UID: \"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"
Nov 02 18:27:53.763298 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.763117    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-p9mpc\" (UID: \"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"
Nov 02 18:27:53.864532 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.864499    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-p9mpc\" (UID: \"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"
Nov 02 18:27:53.864712 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.864610    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-p9mpc\" (UID: \"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"
Nov 02 18:27:53.864712 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.864640    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-2pgkd\" (UniqueName: \"kubernetes.io/projected/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-kube-api-access-2pgkd\") pod \"maas-api-revoke-sub-452b6d7b-p9mpc\" (UID: \"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"
Nov 02 18:27:53.865388 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.865364    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-452b6d7b-p9mpc\" (UID: \"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"
Nov 02 18:27:53.867265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.867243    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-maas-api-xks-ca\") pod \"maas-api-revoke-sub-452b6d7b-p9mpc\" (UID: \"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"
Nov 02 18:27:53.889127 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.889062    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-2pgkd\" (UniqueName: \"kubernetes.io/projected/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-kube-api-access-2pgkd\") pod \"maas-api-revoke-sub-452b6d7b-p9mpc\" (UID: \"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"
Nov 02 18:27:53.960676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:53.960644    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"
Nov 02 18:27:54.260469 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:54.260378    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"]
Nov 02 18:27:54.261347 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:27:54.261316    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod8b1a1bfa_b8ef_41d7_9e02_81817b8c2883.slice/crio-f69060a478eae19a63aacd8e00470027099ba5a4c850c3fdcd56a7ccb6097a5f WatchSource:0}: Error finding container f69060a478eae19a63aacd8e00470027099ba5a4c850c3fdcd56a7ccb6097a5f: Status 404 returned error can't find the container with id f69060a478eae19a63aacd8e00470027099ba5a4c850c3fdcd56a7ccb6097a5f
Nov 02 18:27:54.814448 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:54.814407    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc" event={"ID":"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883","Type":"ContainerStarted","Data":"41e22f6f9c879beca7ade73ce0e544b60e858a77e4ce565e74be6402327b8cd1"}
Nov 02 18:27:54.814959 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:54.814476    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc" event={"ID":"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883","Type":"ContainerStarted","Data":"f69060a478eae19a63aacd8e00470027099ba5a4c850c3fdcd56a7ccb6097a5f"}
Nov 02 18:27:54.857564 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:54.857452    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc" podStartSLOduration=1.857438793 podStartE2EDuration="1.857438793s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:27:53 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:27:54.856912228 +0000 UTC m=+3491.347975411" watchObservedRunningTime="2026-11-02 18:27:54.857438793 +0000 UTC m=+3491.348501993"
Nov 02 18:27:55.824464 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:55.824428    4456 generic.go:356] "Generic (PLEG): container finished" podID="8b1a1bfa-b8ef-41d7-9e02-81817b8c2883" containerID="41e22f6f9c879beca7ade73ce0e544b60e858a77e4ce565e74be6402327b8cd1" exitCode=0
Nov 02 18:27:55.824923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:55.824524    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc" event={"ID":"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883","Type":"ContainerDied","Data":"41e22f6f9c879beca7ade73ce0e544b60e858a77e4ce565e74be6402327b8cd1"}
Nov 02 18:27:57.368234 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:57.368200    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"
Nov 02 18:27:57.446242 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:57.446201    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-maas-api-openshift-ca\") pod \"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883\" (UID: \"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883\") "
Nov 02 18:27:57.448155 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:57.448127    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-maas-api-openshift-ca" pod "8b1a1bfa-b8ef-41d7-9e02-81817b8c2883" (UID: "8b1a1bfa-b8ef-41d7-9e02-81817b8c2883"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:27:57.448535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:57.448301    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-kube-api-access-2pgkd\" (UniqueName: \"kubernetes.io/projected/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-kube-api-access-2pgkd\") pod \"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883\" (UID: \"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883\") "
Nov 02 18:27:57.448752 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:57.448731    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-maas-api-xks-ca\") pod \"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883\" (UID: \"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883\") "
Nov 02 18:27:57.455489 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:57.455468    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:57.461749 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:57.461727    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-maas-api-xks-ca" pod "8b1a1bfa-b8ef-41d7-9e02-81817b8c2883" (UID: "8b1a1bfa-b8ef-41d7-9e02-81817b8c2883"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:27:57.464594 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:57.464556    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-kube-api-access-2pgkd" pod "8b1a1bfa-b8ef-41d7-9e02-81817b8c2883" (UID: "8b1a1bfa-b8ef-41d7-9e02-81817b8c2883"). InnerVolumeSpecName "kube-api-access-2pgkd". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:27:57.556679 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:57.556568    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2pgkd\" (UniqueName: \"kubernetes.io/projected/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-kube-api-access-2pgkd\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:57.556679 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:57.556621    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:27:57.752459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:57.751938    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lrssd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:27:57.752459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:57.752008    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd" podUID="9d114747-c309-445d-8159-5e532bede175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:27:57.866535 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:57.866442    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc" event={"ID":"8b1a1bfa-b8ef-41d7-9e02-81817b8c2883","Type":"ContainerDied","Data":"f69060a478eae19a63aacd8e00470027099ba5a4c850c3fdcd56a7ccb6097a5f"}
Nov 02 18:27:57.866913 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:57.866891    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="f69060a478eae19a63aacd8e00470027099ba5a4c850c3fdcd56a7ccb6097a5f"
Nov 02 18:27:57.867088 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:27:57.866795    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"
Nov 02 18:28:04.977856 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:04.977785    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:09.675260 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:09.675223    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"]
Nov 02 18:28:09.765395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:09.765360    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:13.372471 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.372421    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"]
Nov 02 18:28:13.373082 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.373062    4456 cpu_manager.go:414] "containerMap: removing container" podUID="8b1a1bfa-b8ef-41d7-9e02-81817b8c2883" containerName="revoke-keys"
Nov 02 18:28:13.373175 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.373085    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="8b1a1bfa-b8ef-41d7-9e02-81817b8c2883" containerName="revoke-keys"
Nov 02 18:28:13.373249 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.373234    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="8b1a1bfa-b8ef-41d7-9e02-81817b8c2883" containerName="revoke-keys"
Nov 02 18:28:13.376932 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.376909    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"
Nov 02 18:28:13.408836 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.398700    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"]
Nov 02 18:28:13.409852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.409302    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:28:13.410012 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.409349    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:28:13.410125 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.409467    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:28:13.410221 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.409530    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:28:13.449997 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.449967    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9548e1c9-9efe-4459-ac94-ae198fb8fe99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-s82hw\" (UID: \"9548e1c9-9efe-4459-ac94-ae198fb8fe99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"
Nov 02 18:28:13.450306 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.450284    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9548e1c9-9efe-4459-ac94-ae198fb8fe99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-s82hw\" (UID: \"9548e1c9-9efe-4459-ac94-ae198fb8fe99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"
Nov 02 18:28:13.450520 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.450502    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-d99nb\" (UniqueName: \"kubernetes.io/projected/9548e1c9-9efe-4459-ac94-ae198fb8fe99-kube-api-access-d99nb\") pod \"maas-api-revoke-sub-3e31a506-s82hw\" (UID: \"9548e1c9-9efe-4459-ac94-ae198fb8fe99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"
Nov 02 18:28:13.553840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.553414    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9548e1c9-9efe-4459-ac94-ae198fb8fe99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-s82hw\" (UID: \"9548e1c9-9efe-4459-ac94-ae198fb8fe99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"
Nov 02 18:28:13.553840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.553490    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9548e1c9-9efe-4459-ac94-ae198fb8fe99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-s82hw\" (UID: \"9548e1c9-9efe-4459-ac94-ae198fb8fe99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"
Nov 02 18:28:13.553840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.553528    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-d99nb\" (UniqueName: \"kubernetes.io/projected/9548e1c9-9efe-4459-ac94-ae198fb8fe99-kube-api-access-d99nb\") pod \"maas-api-revoke-sub-3e31a506-s82hw\" (UID: \"9548e1c9-9efe-4459-ac94-ae198fb8fe99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"
Nov 02 18:28:13.554726 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.554695    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9548e1c9-9efe-4459-ac94-ae198fb8fe99-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-3e31a506-s82hw\" (UID: \"9548e1c9-9efe-4459-ac94-ae198fb8fe99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"
Nov 02 18:28:13.561258 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.561233    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9548e1c9-9efe-4459-ac94-ae198fb8fe99-maas-api-xks-ca\") pod \"maas-api-revoke-sub-3e31a506-s82hw\" (UID: \"9548e1c9-9efe-4459-ac94-ae198fb8fe99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"
Nov 02 18:28:13.577783 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.577758    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-d99nb\" (UniqueName: \"kubernetes.io/projected/9548e1c9-9efe-4459-ac94-ae198fb8fe99-kube-api-access-d99nb\") pod \"maas-api-revoke-sub-3e31a506-s82hw\" (UID: \"9548e1c9-9efe-4459-ac94-ae198fb8fe99\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"
Nov 02 18:28:13.712841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:13.711986    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"
Nov 02 18:28:14.153890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:14.153645    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"]
Nov 02 18:28:14.162209 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:28:14.157770    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod9548e1c9_9efe_4459_ac94_ae198fb8fe99.slice/crio-2bc886a664413a805a86e9bdd5713bca11babf6ba75c2f45200e0d38339184c1 WatchSource:0}: Error finding container 2bc886a664413a805a86e9bdd5713bca11babf6ba75c2f45200e0d38339184c1: Status 404 returned error can't find the container with id 2bc886a664413a805a86e9bdd5713bca11babf6ba75c2f45200e0d38339184c1
Nov 02 18:28:14.241840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:14.226545    4456 scope.go:122] "RemoveContainer" containerID="f5dbb073c90644db2950511066eaf61dc4bc1b3259c75f880fc2ff6bac360a33"
Nov 02 18:28:14.259346 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:14.259234    4456 scope.go:122] "RemoveContainer" containerID="a1c7abedd163731d1dea28fe7bb0a243d7c58b3ab5830334205e61dc72b66a1d"
Nov 02 18:28:14.268632 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:14.268611    4456 scope.go:122] "RemoveContainer" containerID="9c62a377bcd5dd7985b529101a480a904b85752a77da4dd783c7c25f86f11bb1"
Nov 02 18:28:15.042722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:15.042594    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw" event={"ID":"9548e1c9-9efe-4459-ac94-ae198fb8fe99","Type":"ContainerStarted","Data":"96aef0e60dace19aff710f43029bace1c4404ccb5a8dd28db8141443c21ac8f0"}
Nov 02 18:28:15.042722 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:15.042674    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw" event={"ID":"9548e1c9-9efe-4459-ac94-ae198fb8fe99","Type":"ContainerStarted","Data":"2bc886a664413a805a86e9bdd5713bca11babf6ba75c2f45200e0d38339184c1"}
Nov 02 18:28:15.086174 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:15.086125    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw" podStartSLOduration=2.086109593 podStartE2EDuration="2.086109593s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:28:13 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:28:15.084205012 +0000 UTC m=+3511.575268190" watchObservedRunningTime="2026-11-02 18:28:15.086109593 +0000 UTC m=+3511.577172754"
Nov 02 18:28:16.048347 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:16.048311    4456 generic.go:356] "Generic (PLEG): container finished" podID="9548e1c9-9efe-4459-ac94-ae198fb8fe99" containerID="96aef0e60dace19aff710f43029bace1c4404ccb5a8dd28db8141443c21ac8f0" exitCode=0
Nov 02 18:28:16.048746 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:16.048393    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw" event={"ID":"9548e1c9-9efe-4459-ac94-ae198fb8fe99","Type":"ContainerDied","Data":"96aef0e60dace19aff710f43029bace1c4404ccb5a8dd28db8141443c21ac8f0"}
Nov 02 18:28:16.169272 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:28:16.169175    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:28:17.197084 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:17.197057    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"
Nov 02 18:28:17.243772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:17.243745    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/9548e1c9-9efe-4459-ac94-ae198fb8fe99-kube-api-access-d99nb\" (UniqueName: \"kubernetes.io/projected/9548e1c9-9efe-4459-ac94-ae198fb8fe99-kube-api-access-d99nb\") pod \"9548e1c9-9efe-4459-ac94-ae198fb8fe99\" (UID: \"9548e1c9-9efe-4459-ac94-ae198fb8fe99\") "
Nov 02 18:28:17.243982 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:17.243858    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/9548e1c9-9efe-4459-ac94-ae198fb8fe99-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9548e1c9-9efe-4459-ac94-ae198fb8fe99-maas-api-openshift-ca\") pod \"9548e1c9-9efe-4459-ac94-ae198fb8fe99\" (UID: \"9548e1c9-9efe-4459-ac94-ae198fb8fe99\") "
Nov 02 18:28:17.243982 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:17.243921    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/9548e1c9-9efe-4459-ac94-ae198fb8fe99-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9548e1c9-9efe-4459-ac94-ae198fb8fe99-maas-api-xks-ca\") pod \"9548e1c9-9efe-4459-ac94-ae198fb8fe99\" (UID: \"9548e1c9-9efe-4459-ac94-ae198fb8fe99\") "
Nov 02 18:28:17.244257 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:17.244234    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/9548e1c9-9efe-4459-ac94-ae198fb8fe99-maas-api-openshift-ca" pod "9548e1c9-9efe-4459-ac94-ae198fb8fe99" (UID: "9548e1c9-9efe-4459-ac94-ae198fb8fe99"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:28:17.246485 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:17.246454    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/9548e1c9-9efe-4459-ac94-ae198fb8fe99-kube-api-access-d99nb" pod "9548e1c9-9efe-4459-ac94-ae198fb8fe99" (UID: "9548e1c9-9efe-4459-ac94-ae198fb8fe99"). InnerVolumeSpecName "kube-api-access-d99nb". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:17.246611 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:17.246524    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/9548e1c9-9efe-4459-ac94-ae198fb8fe99-maas-api-xks-ca" pod "9548e1c9-9efe-4459-ac94-ae198fb8fe99" (UID: "9548e1c9-9efe-4459-ac94-ae198fb8fe99"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:28:17.345601 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:17.345503    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-d99nb\" (UniqueName: \"kubernetes.io/projected/9548e1c9-9efe-4459-ac94-ae198fb8fe99-kube-api-access-d99nb\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:17.345601 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:17.345539    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/9548e1c9-9efe-4459-ac94-ae198fb8fe99-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:17.345601 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:17.345556    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/9548e1c9-9efe-4459-ac94-ae198fb8fe99-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:18.060828 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:18.060790    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"
Nov 02 18:28:18.060993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:18.060859    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw" event={"ID":"9548e1c9-9efe-4459-ac94-ae198fb8fe99","Type":"ContainerDied","Data":"2bc886a664413a805a86e9bdd5713bca11babf6ba75c2f45200e0d38339184c1"}
Nov 02 18:28:18.060993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:18.060906    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="2bc886a664413a805a86e9bdd5713bca11babf6ba75c2f45200e0d38339184c1"
Nov 02 18:28:24.775479 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:24.775437    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw" podUID="787de320-4dca-4811-affc-37617eba4965" containerName="main" containerID="cri-o://8e6acff93f0e12722674839db84d0cf750d39430ef33731fc85190ca00990fc5" gracePeriod=45
Nov 02 18:28:24.781461 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:24.781423    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw" podUID="787de320-4dca-4811-affc-37617eba4965" containerName="main" probeResult="failure" output="Get \"https://10.134.1.50:8000/ready\": EOF"
Nov 02 18:28:25.037152 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.037085    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:28:25.101852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.099582    4456 generic.go:356] "Generic (PLEG): container finished" podID="787de320-4dca-4811-affc-37617eba4965" containerID="8e6acff93f0e12722674839db84d0cf750d39430ef33731fc85190ca00990fc5" exitCode=0
Nov 02 18:28:25.101852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.099662    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"
Nov 02 18:28:25.101852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.099684    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw" event={"ID":"787de320-4dca-4811-affc-37617eba4965","Type":"ContainerDied","Data":"8e6acff93f0e12722674839db84d0cf750d39430ef33731fc85190ca00990fc5"}
Nov 02 18:28:25.101852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.099732    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw" event={"ID":"787de320-4dca-4811-affc-37617eba4965","Type":"ContainerDied","Data":"dcc83ea6c51382b2629afc76858beff0472256f98244b458e6f2c001150fc07d"}
Nov 02 18:28:25.101852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.099755    4456 scope.go:122] "RemoveContainer" containerID="8e6acff93f0e12722674839db84d0cf750d39430ef33731fc85190ca00990fc5"
Nov 02 18:28:25.110528 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.110489    4456 scope.go:122] "RemoveContainer" containerID="8e6acff93f0e12722674839db84d0cf750d39430ef33731fc85190ca00990fc5"
Nov 02 18:28:25.110789 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:28:25.110772    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"8e6acff93f0e12722674839db84d0cf750d39430ef33731fc85190ca00990fc5\": container with ID starting with 8e6acff93f0e12722674839db84d0cf750d39430ef33731fc85190ca00990fc5 not found: ID does not exist" containerID="8e6acff93f0e12722674839db84d0cf750d39430ef33731fc85190ca00990fc5"
Nov 02 18:28:25.110895 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.110802    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"8e6acff93f0e12722674839db84d0cf750d39430ef33731fc85190ca00990fc5"} err="failed to get container status \"8e6acff93f0e12722674839db84d0cf750d39430ef33731fc85190ca00990fc5\": rpc error: code = NotFound desc = could not find container \"8e6acff93f0e12722674839db84d0cf750d39430ef33731fc85190ca00990fc5\": container with ID starting with 8e6acff93f0e12722674839db84d0cf750d39430ef33731fc85190ca00990fc5 not found: ID does not exist"
Nov 02 18:28:25.122737 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.122707    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/787de320-4dca-4811-affc-37617eba4965-tls-certs\" (UniqueName: \"kubernetes.io/secret/787de320-4dca-4811-affc-37617eba4965-tls-certs\") pod \"787de320-4dca-4811-affc-37617eba4965\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") "
Nov 02 18:28:25.122916 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.122752    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/787de320-4dca-4811-affc-37617eba4965-kube-api-access-2mtd8\" (UniqueName: \"kubernetes.io/projected/787de320-4dca-4811-affc-37617eba4965-kube-api-access-2mtd8\") pod \"787de320-4dca-4811-affc-37617eba4965\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") "
Nov 02 18:28:25.122916 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.122784    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-model-cache\") pod \"787de320-4dca-4811-affc-37617eba4965\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") "
Nov 02 18:28:25.123045 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.122914    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-dshm\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-dshm\") pod \"787de320-4dca-4811-affc-37617eba4965\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") "
Nov 02 18:28:25.123045 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.123002    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-tmp-dir\") pod \"787de320-4dca-4811-affc-37617eba4965\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") "
Nov 02 18:28:25.123152 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.123120    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-home\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-home\") pod \"787de320-4dca-4811-affc-37617eba4965\" (UID: \"787de320-4dca-4811-affc-37617eba4965\") "
Nov 02 18:28:25.123855 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.123748    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-tmp-dir" pod "787de320-4dca-4811-affc-37617eba4965" (UID: "787de320-4dca-4811-affc-37617eba4965"). InnerVolumeSpecName "tmp-dir". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:25.123855 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.123761    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-model-cache" pod "787de320-4dca-4811-affc-37617eba4965" (UID: "787de320-4dca-4811-affc-37617eba4965"). InnerVolumeSpecName "model-cache". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:25.123855 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.123782    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-home" pod "787de320-4dca-4811-affc-37617eba4965" (UID: "787de320-4dca-4811-affc-37617eba4965"). InnerVolumeSpecName "home". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:25.126116 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.126091    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-dshm" pod "787de320-4dca-4811-affc-37617eba4965" (UID: "787de320-4dca-4811-affc-37617eba4965"). InnerVolumeSpecName "dshm". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:28:25.126524 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.126495    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/787de320-4dca-4811-affc-37617eba4965-tls-certs" pod "787de320-4dca-4811-affc-37617eba4965" (UID: "787de320-4dca-4811-affc-37617eba4965"). InnerVolumeSpecName "tls-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:28:25.127973 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.127948    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/787de320-4dca-4811-affc-37617eba4965-kube-api-access-2mtd8" pod "787de320-4dca-4811-affc-37617eba4965" (UID: "787de320-4dca-4811-affc-37617eba4965"). InnerVolumeSpecName "kube-api-access-2mtd8". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:25.226842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.226569    4456 reconciler_common.go:299] "Volume detached for volume \"home\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-home\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:25.226842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.226614    4456 reconciler_common.go:299] "Volume detached for volume \"tls-certs\" (UniqueName: \"kubernetes.io/secret/787de320-4dca-4811-affc-37617eba4965-tls-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:25.226842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.226631    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-2mtd8\" (UniqueName: \"kubernetes.io/projected/787de320-4dca-4811-affc-37617eba4965-kube-api-access-2mtd8\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:25.226842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.226646    4456 reconciler_common.go:299] "Volume detached for volume \"model-cache\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-model-cache\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:25.226842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.226659    4456 reconciler_common.go:299] "Volume detached for volume \"dshm\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-dshm\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:25.226842 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.226672    4456 reconciler_common.go:299] "Volume detached for volume \"tmp-dir\" (UniqueName: \"kubernetes.io/empty-dir/787de320-4dca-4811-affc-37617eba4965-tmp-dir\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:25.513102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.513040    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"]
Nov 02 18:28:25.528578 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.528532    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["llm/e2e-same-id-b-d04a7d07-kserve-58cb574f4f-55jpw"]
Nov 02 18:28:25.672297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:25.672247    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="787de320-4dca-4811-affc-37617eba4965" path="/var/lib/kubelet/pods/787de320-4dca-4811-affc-37617eba4965/volumes"
Nov 02 18:28:31.348992 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:31.348941    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-5knvx container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:28:31.350341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:31.350298    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-5knvx" podUID="8ef79a10-292b-42d9-aab0-53bee3e43ed2" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.0.115:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:28:36.842527 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.842487    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"]
Nov 02 18:28:36.843101 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.843082    4456 cpu_manager.go:414] "containerMap: removing container" podUID="9548e1c9-9efe-4459-ac94-ae198fb8fe99" containerName="revoke-keys"
Nov 02 18:28:36.843155 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.843106    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="9548e1c9-9efe-4459-ac94-ae198fb8fe99" containerName="revoke-keys"
Nov 02 18:28:36.843191 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.843171    4456 cpu_manager.go:414] "containerMap: removing container" podUID="787de320-4dca-4811-affc-37617eba4965" containerName="main"
Nov 02 18:28:36.843191 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.843178    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="787de320-4dca-4811-affc-37617eba4965" containerName="main"
Nov 02 18:28:36.843265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.843255    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="787de320-4dca-4811-affc-37617eba4965" containerName="main"
Nov 02 18:28:36.843297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.843271    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="9548e1c9-9efe-4459-ac94-ae198fb8fe99" containerName="revoke-keys"
Nov 02 18:28:36.847672 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.847648    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"
Nov 02 18:28:36.865208 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.865182    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"]
Nov 02 18:28:36.876113 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.876082    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"opendatahub-ca\""
Nov 02 18:28:36.876302 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.876119    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:28:36.876302 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.876127    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:28:36.876439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.876150    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-7flf7\""
Nov 02 18:28:36.911310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.911280    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67106641-ccc8-4c65-9069-d69049611fa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53e082c1-wscp6\" (UID: \"67106641-ccc8-4c65-9069-d69049611fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"
Nov 02 18:28:36.911460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.911315    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-fnn6v\" (UniqueName: \"kubernetes.io/projected/67106641-ccc8-4c65-9069-d69049611fa3-kube-api-access-fnn6v\") pod \"maas-api-revoke-sub-53e082c1-wscp6\" (UID: \"67106641-ccc8-4c65-9069-d69049611fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"
Nov 02 18:28:36.911460 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:36.911341    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67106641-ccc8-4c65-9069-d69049611fa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53e082c1-wscp6\" (UID: \"67106641-ccc8-4c65-9069-d69049611fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"
Nov 02 18:28:37.012205 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:37.012173    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67106641-ccc8-4c65-9069-d69049611fa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53e082c1-wscp6\" (UID: \"67106641-ccc8-4c65-9069-d69049611fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"
Nov 02 18:28:37.012374 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:37.012210    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-fnn6v\" (UniqueName: \"kubernetes.io/projected/67106641-ccc8-4c65-9069-d69049611fa3-kube-api-access-fnn6v\") pod \"maas-api-revoke-sub-53e082c1-wscp6\" (UID: \"67106641-ccc8-4c65-9069-d69049611fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"
Nov 02 18:28:37.012374 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:37.012238    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67106641-ccc8-4c65-9069-d69049611fa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53e082c1-wscp6\" (UID: \"67106641-ccc8-4c65-9069-d69049611fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"
Nov 02 18:28:37.012829 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:37.012781    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67106641-ccc8-4c65-9069-d69049611fa3-maas-api-openshift-ca\") pod \"maas-api-revoke-sub-53e082c1-wscp6\" (UID: \"67106641-ccc8-4c65-9069-d69049611fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"
Nov 02 18:28:37.014809 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:37.014791    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67106641-ccc8-4c65-9069-d69049611fa3-maas-api-xks-ca\") pod \"maas-api-revoke-sub-53e082c1-wscp6\" (UID: \"67106641-ccc8-4c65-9069-d69049611fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"
Nov 02 18:28:37.036742 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:37.036711    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-fnn6v\" (UniqueName: \"kubernetes.io/projected/67106641-ccc8-4c65-9069-d69049611fa3-kube-api-access-fnn6v\") pod \"maas-api-revoke-sub-53e082c1-wscp6\" (UID: \"67106641-ccc8-4c65-9069-d69049611fa3\") " pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"
Nov 02 18:28:37.161544 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:37.161471    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"
Nov 02 18:28:37.457363 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:37.457334    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"]
Nov 02 18:28:37.458469 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:28:37.458439    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod67106641_ccc8_4c65_9069_d69049611fa3.slice/crio-6653ac91443c98bec0a6984cb19008d92596991b192dae7ca913c98d04635421 WatchSource:0}: Error finding container 6653ac91443c98bec0a6984cb19008d92596991b192dae7ca913c98d04635421: Status 404 returned error can't find the container with id 6653ac91443c98bec0a6984cb19008d92596991b192dae7ca913c98d04635421
Nov 02 18:28:38.031567 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:38.031535    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"]
Nov 02 18:28:38.044678 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:38.044602    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3d4156f1-4krw5"]
Nov 02 18:28:38.210332 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:38.210288    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6" event={"ID":"67106641-ccc8-4c65-9069-d69049611fa3","Type":"ContainerStarted","Data":"0bcd5a1446301e0d787b59e53ca66408a3abafdc958820c58fffcf2c40b4a210"}
Nov 02 18:28:38.210514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:38.210348    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6" event={"ID":"67106641-ccc8-4c65-9069-d69049611fa3","Type":"ContainerStarted","Data":"6653ac91443c98bec0a6984cb19008d92596991b192dae7ca913c98d04635421"}
Nov 02 18:28:38.262715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:38.255871    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6" podStartSLOduration=2.255853246 podStartE2EDuration="2.255853246s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:28:36 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:28:38.252069655 +0000 UTC m=+3534.743132827" watchObservedRunningTime="2026-11-02 18:28:38.255853246 +0000 UTC m=+3534.746916408"
Nov 02 18:28:39.216872 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:39.216833    4456 generic.go:356] "Generic (PLEG): container finished" podID="67106641-ccc8-4c65-9069-d69049611fa3" containerID="0bcd5a1446301e0d787b59e53ca66408a3abafdc958820c58fffcf2c40b4a210" exitCode=0
Nov 02 18:28:39.217284 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:39.216913    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6" event={"ID":"67106641-ccc8-4c65-9069-d69049611fa3","Type":"ContainerDied","Data":"0bcd5a1446301e0d787b59e53ca66408a3abafdc958820c58fffcf2c40b4a210"}
Nov 02 18:28:39.613075 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:39.612999    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a336e820-a559-4d4f-b11a-c2d2da68254a" path="/var/lib/kubelet/pods/a336e820-a559-4d4f-b11a-c2d2da68254a/volumes"
Nov 02 18:28:40.007505 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:40.007472    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:40.354273 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:40.354246    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"
Nov 02 18:28:40.443588 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:40.443555    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/67106641-ccc8-4c65-9069-d69049611fa3-maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67106641-ccc8-4c65-9069-d69049611fa3-maas-api-xks-ca\") pod \"67106641-ccc8-4c65-9069-d69049611fa3\" (UID: \"67106641-ccc8-4c65-9069-d69049611fa3\") "
Nov 02 18:28:40.443747 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:40.443614    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/67106641-ccc8-4c65-9069-d69049611fa3-maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67106641-ccc8-4c65-9069-d69049611fa3-maas-api-openshift-ca\") pod \"67106641-ccc8-4c65-9069-d69049611fa3\" (UID: \"67106641-ccc8-4c65-9069-d69049611fa3\") "
Nov 02 18:28:40.443747 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:40.443676    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/67106641-ccc8-4c65-9069-d69049611fa3-kube-api-access-fnn6v\" (UniqueName: \"kubernetes.io/projected/67106641-ccc8-4c65-9069-d69049611fa3-kube-api-access-fnn6v\") pod \"67106641-ccc8-4c65-9069-d69049611fa3\" (UID: \"67106641-ccc8-4c65-9069-d69049611fa3\") "
Nov 02 18:28:40.444069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:40.444028    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/67106641-ccc8-4c65-9069-d69049611fa3-maas-api-openshift-ca" pod "67106641-ccc8-4c65-9069-d69049611fa3" (UID: "67106641-ccc8-4c65-9069-d69049611fa3"). InnerVolumeSpecName "maas-api-openshift-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:28:40.445887 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:40.445868    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/67106641-ccc8-4c65-9069-d69049611fa3-maas-api-xks-ca" pod "67106641-ccc8-4c65-9069-d69049611fa3" (UID: "67106641-ccc8-4c65-9069-d69049611fa3"). InnerVolumeSpecName "maas-api-xks-ca". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:28:40.445987 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:40.445972    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/67106641-ccc8-4c65-9069-d69049611fa3-kube-api-access-fnn6v" pod "67106641-ccc8-4c65-9069-d69049611fa3" (UID: "67106641-ccc8-4c65-9069-d69049611fa3"). InnerVolumeSpecName "kube-api-access-fnn6v". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:40.545308 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:40.545227    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-xks-ca\" (UniqueName: \"kubernetes.io/secret/67106641-ccc8-4c65-9069-d69049611fa3-maas-api-xks-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:40.545308 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:40.545258    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-openshift-ca\" (UniqueName: \"kubernetes.io/configmap/67106641-ccc8-4c65-9069-d69049611fa3-maas-api-openshift-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:40.545308 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:40.545275    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-fnn6v\" (UniqueName: \"kubernetes.io/projected/67106641-ccc8-4c65-9069-d69049611fa3-kube-api-access-fnn6v\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:41.228638 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:41.228608    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"
Nov 02 18:28:41.228841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:41.228652    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6" event={"ID":"67106641-ccc8-4c65-9069-d69049611fa3","Type":"ContainerDied","Data":"6653ac91443c98bec0a6984cb19008d92596991b192dae7ca913c98d04635421"}
Nov 02 18:28:41.228841 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:41.228705    4456 pod_container_deletor.go:80] "Container not found in pod's containers" containerID="6653ac91443c98bec0a6984cb19008d92596991b192dae7ca913c98d04635421"
Nov 02 18:28:46.164295 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:28:46.164260    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:28:49.188051 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.188021    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"]
Nov 02 18:28:49.188509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.188406    4456 cpu_manager.go:414] "containerMap: removing container" podUID="67106641-ccc8-4c65-9069-d69049611fa3" containerName="revoke-keys"
Nov 02 18:28:49.188509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.188417    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="67106641-ccc8-4c65-9069-d69049611fa3" containerName="revoke-keys"
Nov 02 18:28:49.188583 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.188516    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="67106641-ccc8-4c65-9069-d69049611fa3" containerName="revoke-keys"
Nov 02 18:28:49.191529 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.191512    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:49.209529 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.209502    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-metrics-tls\""
Nov 02 18:28:49.212167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.212140    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-64678b9774-s75tm"]
Nov 02 18:28:49.216084 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.216061    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:49.223187 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.223163    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-serving-cert\""
Nov 02 18:28:49.225850 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.225829    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-dockercfg-gml5d\""
Nov 02 18:28:49.226003 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.225857    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-metrics-certs\") pod \"maas-api-667c5f9d54-2zkht\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:49.226227 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.226202    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-maas-api-tls\") pod \"maas-api-667c5f9d54-2zkht\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:49.226324 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.226276    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-openshift-service-ca\") pod \"maas-api-667c5f9d54-2zkht\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:49.226392 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.226360    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-jq469\" (UniqueName: \"kubernetes.io/projected/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-kube-api-access-jq469\") pod \"maas-api-667c5f9d54-2zkht\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:49.229076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.229059    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"]
Nov 02 18:28:49.248508 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.248463    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64678b9774-s75tm"]
Nov 02 18:28:49.269478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.269444    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"openshift-service-ca.crt\""
Nov 02 18:28:49.269659 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.269561    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"odh-ai-gateway-infra\"/\"kube-root-ca.crt\""
Nov 02 18:28:49.327232 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.327198    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/522b0328-e5fe-4a99-8c63-39635d795116-maas-api-tls\") pod \"maas-api-64678b9774-s75tm\" (UID: \"522b0328-e5fe-4a99-8c63-39635d795116\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:49.327423 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.327283    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-metrics-certs\") pod \"maas-api-667c5f9d54-2zkht\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:49.327423 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.327322    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/522b0328-e5fe-4a99-8c63-39635d795116-openshift-service-ca\") pod \"maas-api-64678b9774-s75tm\" (UID: \"522b0328-e5fe-4a99-8c63-39635d795116\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:49.327423 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.327381    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-maas-api-tls\") pod \"maas-api-667c5f9d54-2zkht\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:49.327599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.327548    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-openshift-service-ca\") pod \"maas-api-667c5f9d54-2zkht\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:49.327656 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.327605    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-jq469\" (UniqueName: \"kubernetes.io/projected/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-kube-api-access-jq469\") pod \"maas-api-667c5f9d54-2zkht\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:49.327656 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.327639    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-hkf8p\" (UniqueName: \"kubernetes.io/projected/522b0328-e5fe-4a99-8c63-39635d795116-kube-api-access-hkf8p\") pod \"maas-api-64678b9774-s75tm\" (UID: \"522b0328-e5fe-4a99-8c63-39635d795116\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:49.327758 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.327662    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/522b0328-e5fe-4a99-8c63-39635d795116-metrics-certs\") pod \"maas-api-64678b9774-s75tm\" (UID: \"522b0328-e5fe-4a99-8c63-39635d795116\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:49.328236 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.328211    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-openshift-service-ca\") pod \"maas-api-667c5f9d54-2zkht\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:49.330066 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.330044    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-metrics-certs\") pod \"maas-api-667c5f9d54-2zkht\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:49.330269 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.330246    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-maas-api-tls\") pod \"maas-api-667c5f9d54-2zkht\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:49.352384 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.352362    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-jq469\" (UniqueName: \"kubernetes.io/projected/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-kube-api-access-jq469\") pod \"maas-api-667c5f9d54-2zkht\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") " pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:49.428665 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.428634    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-hkf8p\" (UniqueName: \"kubernetes.io/projected/522b0328-e5fe-4a99-8c63-39635d795116-kube-api-access-hkf8p\") pod \"maas-api-64678b9774-s75tm\" (UID: \"522b0328-e5fe-4a99-8c63-39635d795116\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:49.428871 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.428675    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/522b0328-e5fe-4a99-8c63-39635d795116-metrics-certs\") pod \"maas-api-64678b9774-s75tm\" (UID: \"522b0328-e5fe-4a99-8c63-39635d795116\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:49.428871 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.428694    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/522b0328-e5fe-4a99-8c63-39635d795116-maas-api-tls\") pod \"maas-api-64678b9774-s75tm\" (UID: \"522b0328-e5fe-4a99-8c63-39635d795116\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:49.428871 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.428746    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/522b0328-e5fe-4a99-8c63-39635d795116-openshift-service-ca\") pod \"maas-api-64678b9774-s75tm\" (UID: \"522b0328-e5fe-4a99-8c63-39635d795116\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:49.429444 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.429425    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/522b0328-e5fe-4a99-8c63-39635d795116-openshift-service-ca\") pod \"maas-api-64678b9774-s75tm\" (UID: \"522b0328-e5fe-4a99-8c63-39635d795116\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:49.431430 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.431402    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/522b0328-e5fe-4a99-8c63-39635d795116-metrics-certs\") pod \"maas-api-64678b9774-s75tm\" (UID: \"522b0328-e5fe-4a99-8c63-39635d795116\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:49.431521 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.431461    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/522b0328-e5fe-4a99-8c63-39635d795116-maas-api-tls\") pod \"maas-api-64678b9774-s75tm\" (UID: \"522b0328-e5fe-4a99-8c63-39635d795116\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:49.453240 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.453170    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-hkf8p\" (UniqueName: \"kubernetes.io/projected/522b0328-e5fe-4a99-8c63-39635d795116-kube-api-access-hkf8p\") pod \"maas-api-64678b9774-s75tm\" (UID: \"522b0328-e5fe-4a99-8c63-39635d795116\") " pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:49.505320 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.505293    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:49.531651 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.531627    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:49.807986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.807801    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"]
Nov 02 18:28:49.808992 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:28:49.808960    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod7122e48b_5863_4d2c_9ecf_48cebf68f4a7.slice/crio-3e1c6d924a159460b8c7da841053f26fb3b9a053faf8817a07d1b6e2aed13a25 WatchSource:0}: Error finding container 3e1c6d924a159460b8c7da841053f26fb3b9a053faf8817a07d1b6e2aed13a25: Status 404 returned error can't find the container with id 3e1c6d924a159460b8c7da841053f26fb3b9a053faf8817a07d1b6e2aed13a25
Nov 02 18:28:49.810375 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.810355    4456 provider.go:93] Refreshing cache for provider: *credentialprovider.defaultDockerConfigProvider
Nov 02 18:28:49.858582 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:49.858548    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-64678b9774-s75tm"]
Nov 02 18:28:49.859483 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:28:49.859447    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod522b0328_e5fe_4a99_8c63_39635d795116.slice/crio-51f7444d4cd120dfd29507feca9aa3b73da079dc30f66d8d81d3ae0feaa31a63 WatchSource:0}: Error finding container 51f7444d4cd120dfd29507feca9aa3b73da079dc30f66d8d81d3ae0feaa31a63: Status 404 returned error can't find the container with id 51f7444d4cd120dfd29507feca9aa3b73da079dc30f66d8d81d3ae0feaa31a63
Nov 02 18:28:50.290402 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:50.290263    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht" event={"ID":"7122e48b-5863-4d2c-9ecf-48cebf68f4a7","Type":"ContainerStarted","Data":"6c0ceb7f309cbb4ee16b1e9052ded5ad1af97b17c7a5fc21e7a3c64a3df81dca"}
Nov 02 18:28:50.290402 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:50.290339    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht" event={"ID":"7122e48b-5863-4d2c-9ecf-48cebf68f4a7","Type":"ContainerStarted","Data":"3e1c6d924a159460b8c7da841053f26fb3b9a053faf8817a07d1b6e2aed13a25"}
Nov 02 18:28:50.290402 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:50.290370    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:50.294387 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:50.293686    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm" event={"ID":"522b0328-e5fe-4a99-8c63-39635d795116","Type":"ContainerStarted","Data":"2133ec6b3c3c61cf962fa7fba7769b41ef7692ef7f5bb22f0fe18ea5963e6f44"}
Nov 02 18:28:50.294387 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:50.293744    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm" event={"ID":"522b0328-e5fe-4a99-8c63-39635d795116","Type":"ContainerStarted","Data":"51f7444d4cd120dfd29507feca9aa3b73da079dc30f66d8d81d3ae0feaa31a63"}
Nov 02 18:28:50.294387 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:50.294330    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:50.348675 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:50.348625    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht" podStartSLOduration=0.954002894 podStartE2EDuration="1.348611724s" totalImagesPullingTime="394.60883ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:28:50.346370022 +0000 UTC m=+3546.837433182" watchObservedRunningTime="2026-11-02 18:28:50.348611724 +0000 UTC m=+3546.839674885"
Nov 02 18:28:50.391928 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:50.391871    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm" podStartSLOduration=1.025122784 podStartE2EDuration="1.39185762s" totalImagesPullingTime="366.734836ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:28:49 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:28:50.390104818 +0000 UTC m=+3546.881167980" watchObservedRunningTime="2026-11-02 18:28:50.39185762 +0000 UTC m=+3546.882920782"
Nov 02 18:28:56.122030 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:56.121987    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:56.304475 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:56.304444    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:56.624181 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:56.624148    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:56.787509 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:56.787475    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:57.310026 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.309991    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-64678b9774-s75tm"
Nov 02 18:28:57.385973 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.385941    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"]
Nov 02 18:28:57.386233 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.386180    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht" podUID="7122e48b-5863-4d2c-9ecf-48cebf68f4a7" containerName="maas-api" containerID="cri-o://6c0ceb7f309cbb4ee16b1e9052ded5ad1af97b17c7a5fc21e7a3c64a3df81dca" gracePeriod=30
Nov 02 18:28:57.650179 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.650155    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:57.713026 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.713000    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-kube-api-access-jq469\" (UniqueName: \"kubernetes.io/projected/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-kube-api-access-jq469\") pod \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") "
Nov 02 18:28:57.713166 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.713053    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-metrics-certs\" (UniqueName: \"kubernetes.io/secret/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-metrics-certs\") pod \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") "
Nov 02 18:28:57.713166 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.713099    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-openshift-service-ca\") pod \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") "
Nov 02 18:28:57.713246 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.713180    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-maas-api-tls\") pod \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\" (UID: \"7122e48b-5863-4d2c-9ecf-48cebf68f4a7\") "
Nov 02 18:28:57.713630 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.713598    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-openshift-service-ca" pod "7122e48b-5863-4d2c-9ecf-48cebf68f4a7" (UID: "7122e48b-5863-4d2c-9ecf-48cebf68f4a7"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:28:57.715478 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.715455    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-kube-api-access-jq469" pod "7122e48b-5863-4d2c-9ecf-48cebf68f4a7" (UID: "7122e48b-5863-4d2c-9ecf-48cebf68f4a7"). InnerVolumeSpecName "kube-api-access-jq469". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:28:57.715958 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.715938    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-metrics-certs" pod "7122e48b-5863-4d2c-9ecf-48cebf68f4a7" (UID: "7122e48b-5863-4d2c-9ecf-48cebf68f4a7"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:28:57.716045 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.715955    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-maas-api-tls" pod "7122e48b-5863-4d2c-9ecf-48cebf68f4a7" (UID: "7122e48b-5863-4d2c-9ecf-48cebf68f4a7"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:28:57.814322 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.814275    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-jq469\" (UniqueName: \"kubernetes.io/projected/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-kube-api-access-jq469\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:57.814322 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.814308    4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-metrics-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:57.814322 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.814325    4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-openshift-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:57.814322 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:57.814334    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/7122e48b-5863-4d2c-9ecf-48cebf68f4a7-maas-api-tls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:28:58.339455 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:58.339421    4456 generic.go:356] "Generic (PLEG): container finished" podID="7122e48b-5863-4d2c-9ecf-48cebf68f4a7" containerID="6c0ceb7f309cbb4ee16b1e9052ded5ad1af97b17c7a5fc21e7a3c64a3df81dca" exitCode=0
Nov 02 18:28:58.339984 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:58.339466    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"
Nov 02 18:28:58.339984 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:58.339514    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht" event={"ID":"7122e48b-5863-4d2c-9ecf-48cebf68f4a7","Type":"ContainerDied","Data":"6c0ceb7f309cbb4ee16b1e9052ded5ad1af97b17c7a5fc21e7a3c64a3df81dca"}
Nov 02 18:28:58.339984 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:58.339578    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht" event={"ID":"7122e48b-5863-4d2c-9ecf-48cebf68f4a7","Type":"ContainerDied","Data":"3e1c6d924a159460b8c7da841053f26fb3b9a053faf8817a07d1b6e2aed13a25"}
Nov 02 18:28:58.339984 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:58.339609    4456 scope.go:122] "RemoveContainer" containerID="6c0ceb7f309cbb4ee16b1e9052ded5ad1af97b17c7a5fc21e7a3c64a3df81dca"
Nov 02 18:28:58.349474 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:58.349456    4456 scope.go:122] "RemoveContainer" containerID="6c0ceb7f309cbb4ee16b1e9052ded5ad1af97b17c7a5fc21e7a3c64a3df81dca"
Nov 02 18:28:58.349720 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:28:58.349704    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"6c0ceb7f309cbb4ee16b1e9052ded5ad1af97b17c7a5fc21e7a3c64a3df81dca\": container with ID starting with 6c0ceb7f309cbb4ee16b1e9052ded5ad1af97b17c7a5fc21e7a3c64a3df81dca not found: ID does not exist" containerID="6c0ceb7f309cbb4ee16b1e9052ded5ad1af97b17c7a5fc21e7a3c64a3df81dca"
Nov 02 18:28:58.349785 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:58.349729    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"6c0ceb7f309cbb4ee16b1e9052ded5ad1af97b17c7a5fc21e7a3c64a3df81dca"} err="failed to get container status \"6c0ceb7f309cbb4ee16b1e9052ded5ad1af97b17c7a5fc21e7a3c64a3df81dca\": rpc error: code = NotFound desc = could not find container \"6c0ceb7f309cbb4ee16b1e9052ded5ad1af97b17c7a5fc21e7a3c64a3df81dca\": container with ID starting with 6c0ceb7f309cbb4ee16b1e9052ded5ad1af97b17c7a5fc21e7a3c64a3df81dca not found: ID does not exist"
Nov 02 18:28:58.394609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:58.394583    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"]
Nov 02 18:28:58.407759 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:58.407730    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-667c5f9d54-2zkht"]
Nov 02 18:28:59.612454 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:59.612413    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="7122e48b-5863-4d2c-9ecf-48cebf68f4a7" path="/var/lib/kubelet/pods/7122e48b-5863-4d2c-9ecf-48cebf68f4a7/volumes"
Nov 02 18:28:59.747888 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:59.747847    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:28:59.892072 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:28:59.891974    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:01.443925 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.443897    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"]
Nov 02 18:29:01.444329 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.444314    4456 cpu_manager.go:414] "containerMap: removing container" podUID="7122e48b-5863-4d2c-9ecf-48cebf68f4a7" containerName="maas-api"
Nov 02 18:29:01.444329 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.444329    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="7122e48b-5863-4d2c-9ecf-48cebf68f4a7" containerName="maas-api"
Nov 02 18:29:01.444439 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.444393    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="7122e48b-5863-4d2c-9ecf-48cebf68f4a7" containerName="maas-api"
Nov 02 18:29:01.448691 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.448672    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:01.495404 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.495385    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-e2e-ait-ae417f29-serving-cert\""
Nov 02 18:29:01.502505 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.502456    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"]
Nov 02 18:29:01.545609 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.545575    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:01.550808 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.550780    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e31802-3b39-49c4-980c-13d8ff6dbd79-maas-api-tls\") pod \"maas-api-e2e-ait-ae417f29-58d9fb56-djkxn\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:01.550952 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.550890    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-q8gtc\" (UniqueName: \"kubernetes.io/projected/a6e31802-3b39-49c4-980c-13d8ff6dbd79-kube-api-access-q8gtc\") pod \"maas-api-e2e-ait-ae417f29-58d9fb56-djkxn\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:01.550952 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.550938    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e31802-3b39-49c4-980c-13d8ff6dbd79-metrics-certs\") pod \"maas-api-e2e-ait-ae417f29-58d9fb56-djkxn\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:01.551041 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.550956    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e31802-3b39-49c4-980c-13d8ff6dbd79-openshift-service-ca\") pod \"maas-api-e2e-ait-ae417f29-58d9fb56-djkxn\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:01.652155 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.652124    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e31802-3b39-49c4-980c-13d8ff6dbd79-maas-api-tls\") pod \"maas-api-e2e-ait-ae417f29-58d9fb56-djkxn\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:01.652277 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.652220    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-q8gtc\" (UniqueName: \"kubernetes.io/projected/a6e31802-3b39-49c4-980c-13d8ff6dbd79-kube-api-access-q8gtc\") pod \"maas-api-e2e-ait-ae417f29-58d9fb56-djkxn\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:01.652341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.652288    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e31802-3b39-49c4-980c-13d8ff6dbd79-metrics-certs\") pod \"maas-api-e2e-ait-ae417f29-58d9fb56-djkxn\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:01.652341 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.652321    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e31802-3b39-49c4-980c-13d8ff6dbd79-openshift-service-ca\") pod \"maas-api-e2e-ait-ae417f29-58d9fb56-djkxn\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:01.653063 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.653001    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e31802-3b39-49c4-980c-13d8ff6dbd79-openshift-service-ca\") pod \"maas-api-e2e-ait-ae417f29-58d9fb56-djkxn\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:01.654614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.654587    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e31802-3b39-49c4-980c-13d8ff6dbd79-maas-api-tls\") pod \"maas-api-e2e-ait-ae417f29-58d9fb56-djkxn\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:01.654741 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.654724    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e31802-3b39-49c4-980c-13d8ff6dbd79-metrics-certs\") pod \"maas-api-e2e-ait-ae417f29-58d9fb56-djkxn\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:01.678392 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.678369    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-q8gtc\" (UniqueName: \"kubernetes.io/projected/a6e31802-3b39-49c4-980c-13d8ff6dbd79-kube-api-access-q8gtc\") pod \"maas-api-e2e-ait-ae417f29-58d9fb56-djkxn\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") " pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:01.691271 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.691246    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:01.701895 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.701839    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"]
Nov 02 18:29:01.711054 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.711032    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"
Nov 02 18:29:01.725069 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.725047    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"]
Nov 02 18:29:01.726775 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.726748    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"payload-processing-e2e-ait-ae417f29-dockercfg-v8q89\""
Nov 02 18:29:01.726893 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.726789    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-ingress\"/\"payload-processing-plugins-e2e-ait-ae417f29\""
Nov 02 18:29:01.752281 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.752257    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"]
Nov 02 18:29:01.753643 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.753620    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-v47zv\" (UniqueName: \"kubernetes.io/projected/c9d5829b-2121-4ea6-8d95-e0171152e145-kube-api-access-v47zv\") pod \"payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc\" (UID: \"c9d5829b-2121-4ea6-8d95-e0171152e145\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"
Nov 02 18:29:01.753742 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.753677    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d5829b-2121-4ea6-8d95-e0171152e145-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc\" (UID: \"c9d5829b-2121-4ea6-8d95-e0171152e145\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"
Nov 02 18:29:01.756740 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.756719    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"
Nov 02 18:29:01.761226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.761205    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:01.779429 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.779399    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"]
Nov 02 18:29:01.835512 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.835485    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:01.856769 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.855186    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d5829b-2121-4ea6-8d95-e0171152e145-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc\" (UID: \"c9d5829b-2121-4ea6-8d95-e0171152e145\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"
Nov 02 18:29:01.856769 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.855315    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f751f99-6a9d-42cc-a79f-42980a28c682-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5\" (UID: \"6f751f99-6a9d-42cc-a79f-42980a28c682\") " pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"
Nov 02 18:29:01.856769 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.855384    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8m5vx\" (UniqueName: \"kubernetes.io/projected/6f751f99-6a9d-42cc-a79f-42980a28c682-kube-api-access-8m5vx\") pod \"payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5\" (UID: \"6f751f99-6a9d-42cc-a79f-42980a28c682\") " pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"
Nov 02 18:29:01.856769 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.855436    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-v47zv\" (UniqueName: \"kubernetes.io/projected/c9d5829b-2121-4ea6-8d95-e0171152e145-kube-api-access-v47zv\") pod \"payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc\" (UID: \"c9d5829b-2121-4ea6-8d95-e0171152e145\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"
Nov 02 18:29:01.857104 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.856667    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d5829b-2121-4ea6-8d95-e0171152e145-plugins-config-volume\") pod \"payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc\" (UID: \"c9d5829b-2121-4ea6-8d95-e0171152e145\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"
Nov 02 18:29:01.880560 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.880529    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-v47zv\" (UniqueName: \"kubernetes.io/projected/c9d5829b-2121-4ea6-8d95-e0171152e145-kube-api-access-v47zv\") pod \"payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc\" (UID: \"c9d5829b-2121-4ea6-8d95-e0171152e145\") " pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"
Nov 02 18:29:01.957142 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.957058    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f751f99-6a9d-42cc-a79f-42980a28c682-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5\" (UID: \"6f751f99-6a9d-42cc-a79f-42980a28c682\") " pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"
Nov 02 18:29:01.957284 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.957149    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8m5vx\" (UniqueName: \"kubernetes.io/projected/6f751f99-6a9d-42cc-a79f-42980a28c682-kube-api-access-8m5vx\") pod \"payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5\" (UID: \"6f751f99-6a9d-42cc-a79f-42980a28c682\") " pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"
Nov 02 18:29:01.957748 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.957728    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f751f99-6a9d-42cc-a79f-42980a28c682-plugins-config-volume\") pod \"payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5\" (UID: \"6f751f99-6a9d-42cc-a79f-42980a28c682\") " pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"
Nov 02 18:29:01.981481 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:01.981452    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8m5vx\" (UniqueName: \"kubernetes.io/projected/6f751f99-6a9d-42cc-a79f-42980a28c682-kube-api-access-8m5vx\") pod \"payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5\" (UID: \"6f751f99-6a9d-42cc-a79f-42980a28c682\") " pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"
Nov 02 18:29:02.030134 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:02.030109    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"
Nov 02 18:29:02.071540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:02.071499    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"
Nov 02 18:29:02.087378 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:02.087349    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"]
Nov 02 18:29:02.098962 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:29:02.098936    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-poda6e31802_3b39_49c4_980c_13d8ff6dbd79.slice/crio-2fc14c940ad2867e41b1a590a61291a9a119f8062ed0027b6e1b8a0abf7e86da WatchSource:0}: Error finding container 2fc14c940ad2867e41b1a590a61291a9a119f8062ed0027b6e1b8a0abf7e86da: Status 404 returned error can't find the container with id 2fc14c940ad2867e41b1a590a61291a9a119f8062ed0027b6e1b8a0abf7e86da
Nov 02 18:29:02.329656 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:29:02.329619    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podc9d5829b_2121_4ea6_8d95_e0171152e145.slice/crio-b3c0cc0c457173382fbac403cb16833f056b3908f86eb37e90eda4fd9a28a749 WatchSource:0}: Error finding container b3c0cc0c457173382fbac403cb16833f056b3908f86eb37e90eda4fd9a28a749: Status 404 returned error can't find the container with id b3c0cc0c457173382fbac403cb16833f056b3908f86eb37e90eda4fd9a28a749
Nov 02 18:29:02.329782 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:02.329758    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"]
Nov 02 18:29:02.363355 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:02.363323    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc" event={"ID":"c9d5829b-2121-4ea6-8d95-e0171152e145","Type":"ContainerStarted","Data":"b3c0cc0c457173382fbac403cb16833f056b3908f86eb37e90eda4fd9a28a749"}
Nov 02 18:29:02.364874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:02.364846    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn" event={"ID":"a6e31802-3b39-49c4-980c-13d8ff6dbd79","Type":"ContainerStarted","Data":"2fc14c940ad2867e41b1a590a61291a9a119f8062ed0027b6e1b8a0abf7e86da"}
Nov 02 18:29:02.381652 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:02.381628    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"]
Nov 02 18:29:02.382151 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:29:02.382121    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod6f751f99_6a9d_42cc_a79f_42980a28c682.slice/crio-a74b01af13ed5afef21cf4092a9f1518a8de07f0bce4e94ffde0cb9bb2e2fded WatchSource:0}: Error finding container a74b01af13ed5afef21cf4092a9f1518a8de07f0bce4e94ffde0cb9bb2e2fded: Status 404 returned error can't find the container with id a74b01af13ed5afef21cf4092a9f1518a8de07f0bce4e94ffde0cb9bb2e2fded
Nov 02 18:29:03.371524 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:03.371487    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc" event={"ID":"c9d5829b-2121-4ea6-8d95-e0171152e145","Type":"ContainerStarted","Data":"719ea9ea68dbf199d1ceaca0ef7d177ce497270ff893172917bcc10c42f94fbd"}
Nov 02 18:29:03.372022 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:03.371558    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"
Nov 02 18:29:03.373050 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:03.373023    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5" event={"ID":"6f751f99-6a9d-42cc-a79f-42980a28c682","Type":"ContainerStarted","Data":"41f175b3acf2c000fe5bfad2439299274523edd1d6fe89f4e991149993716685"}
Nov 02 18:29:03.373162 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:03.373067    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5" event={"ID":"6f751f99-6a9d-42cc-a79f-42980a28c682","Type":"ContainerStarted","Data":"a74b01af13ed5afef21cf4092a9f1518a8de07f0bce4e94ffde0cb9bb2e2fded"}
Nov 02 18:29:03.373162 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:03.373114    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"
Nov 02 18:29:03.374514 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:03.374490    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn" event={"ID":"a6e31802-3b39-49c4-980c-13d8ff6dbd79","Type":"ContainerStarted","Data":"33b22f6d9507e27f32f92fdaca4d097f3f9791cf66f94d59180ffd30e2e2ad55"}
Nov 02 18:29:03.374586 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:03.374569    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:03.415845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:03.415779    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc" podStartSLOduration=2.415768931 podStartE2EDuration="2.415768931s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:03.413301654 +0000 UTC m=+3559.904364850" watchObservedRunningTime="2026-11-02 18:29:03.415768931 +0000 UTC m=+3559.906832102"
Nov 02 18:29:03.456061 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:03.456018    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5" podStartSLOduration=2.456006314 podStartE2EDuration="2.456006314s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:29:01 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:03.454690143 +0000 UTC m=+3559.945753323" watchObservedRunningTime="2026-11-02 18:29:03.456006314 +0000 UTC m=+3559.947069474"
Nov 02 18:29:03.498310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:03.498268    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn" podStartSLOduration=2.17127043 podStartE2EDuration="2.498254613s" totalImagesPullingTime="326.984183ms" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:29:01 +0000 UTC" imagePullSessionsCount=1 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:03.497975513 +0000 UTC m=+3559.989038685" watchObservedRunningTime="2026-11-02 18:29:03.498254613 +0000 UTC m=+3559.989317774"
Nov 02 18:29:03.661540 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:03.661454    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:05.556322 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:05.556291    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:07.380995 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:07.380962    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:09.280325 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:09.280290    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:09.386156 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:09.386128    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:11.202203 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:11.202109    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:13.147604 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:13.147552    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:14.382788 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:14.382760    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"
Nov 02 18:29:14.383213 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:14.383185    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"
Nov 02 18:29:14.957910 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:14.957855    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:16.165789 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:29:16.165754    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:29:17.093953 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:17.093915    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:18.978996 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:18.978915    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:20.907407 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:20.907360    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:22.662053 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:22.662006    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:24.550501 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:24.550435    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:26.045727 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:26.045697    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"]
Nov 02 18:29:26.076267 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:26.076231    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"]
Nov 02 18:29:26.212890 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:26.212838    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:26.357223 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:26.357128    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:26.557633 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:26.557589    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:27.832161 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:27.832123    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"]
Nov 02 18:29:27.832654 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:27.832377    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn" podUID="a6e31802-3b39-49c4-980c-13d8ff6dbd79" containerName="maas-api" containerID="cri-o://33b22f6d9507e27f32f92fdaca4d097f3f9791cf66f94d59180ffd30e2e2ad55" gracePeriod=30
Nov 02 18:29:28.015646 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.015594    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:28.094285 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.094210    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:28.127299 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.127266    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e31802-3b39-49c4-980c-13d8ff6dbd79-metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e31802-3b39-49c4-980c-13d8ff6dbd79-metrics-certs\") pod \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") "
Nov 02 18:29:28.127445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.127347    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/a6e31802-3b39-49c4-980c-13d8ff6dbd79-openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e31802-3b39-49c4-980c-13d8ff6dbd79-openshift-service-ca\") pod \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") "
Nov 02 18:29:28.127445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.127394    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/secret/a6e31802-3b39-49c4-980c-13d8ff6dbd79-maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e31802-3b39-49c4-980c-13d8ff6dbd79-maas-api-tls\") pod \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") "
Nov 02 18:29:28.127445 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.127424    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/a6e31802-3b39-49c4-980c-13d8ff6dbd79-kube-api-access-q8gtc\" (UniqueName: \"kubernetes.io/projected/a6e31802-3b39-49c4-980c-13d8ff6dbd79-kube-api-access-q8gtc\") pod \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\" (UID: \"a6e31802-3b39-49c4-980c-13d8ff6dbd79\") "
Nov 02 18:29:28.127756 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.127729    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/a6e31802-3b39-49c4-980c-13d8ff6dbd79-openshift-service-ca" pod "a6e31802-3b39-49c4-980c-13d8ff6dbd79" (UID: "a6e31802-3b39-49c4-980c-13d8ff6dbd79"). InnerVolumeSpecName "openshift-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:29:28.129849 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.129796    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e31802-3b39-49c4-980c-13d8ff6dbd79-metrics-certs" pod "a6e31802-3b39-49c4-980c-13d8ff6dbd79" (UID: "a6e31802-3b39-49c4-980c-13d8ff6dbd79"). InnerVolumeSpecName "metrics-certs". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:29:28.129849 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.129839    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/secret/a6e31802-3b39-49c4-980c-13d8ff6dbd79-maas-api-tls" pod "a6e31802-3b39-49c4-980c-13d8ff6dbd79" (UID: "a6e31802-3b39-49c4-980c-13d8ff6dbd79"). InnerVolumeSpecName "maas-api-tls". PluginName "kubernetes.io/secret", VolumeGIDValue ""
Nov 02 18:29:28.131138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.131109    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/a6e31802-3b39-49c4-980c-13d8ff6dbd79-kube-api-access-q8gtc" pod "a6e31802-3b39-49c4-980c-13d8ff6dbd79" (UID: "a6e31802-3b39-49c4-980c-13d8ff6dbd79"). InnerVolumeSpecName "kube-api-access-q8gtc". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:28.228369 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.228331    4456 reconciler_common.go:299] "Volume detached for volume \"openshift-service-ca\" (UniqueName: \"kubernetes.io/configmap/a6e31802-3b39-49c4-980c-13d8ff6dbd79-openshift-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:29:28.228369 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.228366    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-tls\" (UniqueName: \"kubernetes.io/secret/a6e31802-3b39-49c4-980c-13d8ff6dbd79-maas-api-tls\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:29:28.228369 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.228377    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-q8gtc\" (UniqueName: \"kubernetes.io/projected/a6e31802-3b39-49c4-980c-13d8ff6dbd79-kube-api-access-q8gtc\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:29:28.228600 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.228385    4456 reconciler_common.go:299] "Volume detached for volume \"metrics-certs\" (UniqueName: \"kubernetes.io/secret/a6e31802-3b39-49c4-980c-13d8ff6dbd79-metrics-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:29:28.468331 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.468297    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:28.512685 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.512652    4456 generic.go:356] "Generic (PLEG): container finished" podID="a6e31802-3b39-49c4-980c-13d8ff6dbd79" containerID="33b22f6d9507e27f32f92fdaca4d097f3f9791cf66f94d59180ffd30e2e2ad55" exitCode=0
Nov 02 18:29:28.512868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.512700    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"
Nov 02 18:29:28.512868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.512708    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn" event={"ID":"a6e31802-3b39-49c4-980c-13d8ff6dbd79","Type":"ContainerDied","Data":"33b22f6d9507e27f32f92fdaca4d097f3f9791cf66f94d59180ffd30e2e2ad55"}
Nov 02 18:29:28.512868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.512758    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn" event={"ID":"a6e31802-3b39-49c4-980c-13d8ff6dbd79","Type":"ContainerDied","Data":"2fc14c940ad2867e41b1a590a61291a9a119f8062ed0027b6e1b8a0abf7e86da"}
Nov 02 18:29:28.512868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.512779    4456 scope.go:122] "RemoveContainer" containerID="33b22f6d9507e27f32f92fdaca4d097f3f9791cf66f94d59180ffd30e2e2ad55"
Nov 02 18:29:28.523421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.523402    4456 scope.go:122] "RemoveContainer" containerID="33b22f6d9507e27f32f92fdaca4d097f3f9791cf66f94d59180ffd30e2e2ad55"
Nov 02 18:29:28.523708 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:29:28.523689    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"33b22f6d9507e27f32f92fdaca4d097f3f9791cf66f94d59180ffd30e2e2ad55\": container with ID starting with 33b22f6d9507e27f32f92fdaca4d097f3f9791cf66f94d59180ffd30e2e2ad55 not found: ID does not exist" containerID="33b22f6d9507e27f32f92fdaca4d097f3f9791cf66f94d59180ffd30e2e2ad55"
Nov 02 18:29:28.523760 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.523717    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"33b22f6d9507e27f32f92fdaca4d097f3f9791cf66f94d59180ffd30e2e2ad55"} err="failed to get container status \"33b22f6d9507e27f32f92fdaca4d097f3f9791cf66f94d59180ffd30e2e2ad55\": rpc error: code = NotFound desc = could not find container \"33b22f6d9507e27f32f92fdaca4d097f3f9791cf66f94d59180ffd30e2e2ad55\": container with ID starting with 33b22f6d9507e27f32f92fdaca4d097f3f9791cf66f94d59180ffd30e2e2ad55 not found: ID does not exist"
Nov 02 18:29:28.570056 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.570020    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"]
Nov 02 18:29:28.584602 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.584560    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-e2e-ait-ae417f29-58d9fb56-djkxn"]
Nov 02 18:29:28.607968 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:28.607940    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:29.612395 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:29.612361    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="a6e31802-3b39-49c4-980c-13d8ff6dbd79" path="/var/lib/kubelet/pods/a6e31802-3b39-49c4-980c-13d8ff6dbd79/volumes"
Nov 02 18:29:31.046685 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.046642    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5" podUID="6f751f99-6a9d-42cc-a79f-42980a28c682" containerName="payload-processing" containerID="cri-o://41f175b3acf2c000fe5bfad2439299274523edd1d6fe89f4e991149993716685" gracePeriod=25
Nov 02 18:29:31.077451 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.077398    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc" podUID="c9d5829b-2121-4ea6-8d95-e0171152e145" containerName="payload-pre-processing" containerID="cri-o://719ea9ea68dbf199d1ceaca0ef7d177ce497270ff893172917bcc10c42f94fbd" gracePeriod=25
Nov 02 18:29:31.337773 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.337735    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"
Nov 02 18:29:31.358923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.358902    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/6f751f99-6a9d-42cc-a79f-42980a28c682-kube-api-access-8m5vx\" (UniqueName: \"kubernetes.io/projected/6f751f99-6a9d-42cc-a79f-42980a28c682-kube-api-access-8m5vx\") pod \"6f751f99-6a9d-42cc-a79f-42980a28c682\" (UID: \"6f751f99-6a9d-42cc-a79f-42980a28c682\") "
Nov 02 18:29:31.359049 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.358952    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/6f751f99-6a9d-42cc-a79f-42980a28c682-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f751f99-6a9d-42cc-a79f-42980a28c682-plugins-config-volume\") pod \"6f751f99-6a9d-42cc-a79f-42980a28c682\" (UID: \"6f751f99-6a9d-42cc-a79f-42980a28c682\") "
Nov 02 18:29:31.359319 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.359299    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/6f751f99-6a9d-42cc-a79f-42980a28c682-plugins-config-volume" pod "6f751f99-6a9d-42cc-a79f-42980a28c682" (UID: "6f751f99-6a9d-42cc-a79f-42980a28c682"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:29:31.359923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.359897    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"
Nov 02 18:29:31.360988 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.360968    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/6f751f99-6a9d-42cc-a79f-42980a28c682-kube-api-access-8m5vx" pod "6f751f99-6a9d-42cc-a79f-42980a28c682" (UID: "6f751f99-6a9d-42cc-a79f-42980a28c682"). InnerVolumeSpecName "kube-api-access-8m5vx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:31.460081 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.460038    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/c9d5829b-2121-4ea6-8d95-e0171152e145-plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d5829b-2121-4ea6-8d95-e0171152e145-plugins-config-volume\") pod \"c9d5829b-2121-4ea6-8d95-e0171152e145\" (UID: \"c9d5829b-2121-4ea6-8d95-e0171152e145\") "
Nov 02 18:29:31.460262 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.460243    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/c9d5829b-2121-4ea6-8d95-e0171152e145-kube-api-access-v47zv\" (UniqueName: \"kubernetes.io/projected/c9d5829b-2121-4ea6-8d95-e0171152e145-kube-api-access-v47zv\") pod \"c9d5829b-2121-4ea6-8d95-e0171152e145\" (UID: \"c9d5829b-2121-4ea6-8d95-e0171152e145\") "
Nov 02 18:29:31.460386 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.460364    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/c9d5829b-2121-4ea6-8d95-e0171152e145-plugins-config-volume" pod "c9d5829b-2121-4ea6-8d95-e0171152e145" (UID: "c9d5829b-2121-4ea6-8d95-e0171152e145"). InnerVolumeSpecName "plugins-config-volume". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:29:31.460525 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.460511    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/c9d5829b-2121-4ea6-8d95-e0171152e145-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:29:31.460589 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.460528    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8m5vx\" (UniqueName: \"kubernetes.io/projected/6f751f99-6a9d-42cc-a79f-42980a28c682-kube-api-access-8m5vx\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:29:31.460589 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.460538    4456 reconciler_common.go:299] "Volume detached for volume \"plugins-config-volume\" (UniqueName: \"kubernetes.io/configmap/6f751f99-6a9d-42cc-a79f-42980a28c682-plugins-config-volume\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:29:31.462859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.462832    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/c9d5829b-2121-4ea6-8d95-e0171152e145-kube-api-access-v47zv" pod "c9d5829b-2121-4ea6-8d95-e0171152e145" (UID: "c9d5829b-2121-4ea6-8d95-e0171152e145"). InnerVolumeSpecName "kube-api-access-v47zv". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:29:31.530802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.530765    4456 generic.go:356] "Generic (PLEG): container finished" podID="c9d5829b-2121-4ea6-8d95-e0171152e145" containerID="719ea9ea68dbf199d1ceaca0ef7d177ce497270ff893172917bcc10c42f94fbd" exitCode=0
Nov 02 18:29:31.531005 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.530852    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"
Nov 02 18:29:31.531005 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.530857    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc" event={"ID":"c9d5829b-2121-4ea6-8d95-e0171152e145","Type":"ContainerDied","Data":"719ea9ea68dbf199d1ceaca0ef7d177ce497270ff893172917bcc10c42f94fbd"}
Nov 02 18:29:31.531005 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.530929    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc" event={"ID":"c9d5829b-2121-4ea6-8d95-e0171152e145","Type":"ContainerDied","Data":"b3c0cc0c457173382fbac403cb16833f056b3908f86eb37e90eda4fd9a28a749"}
Nov 02 18:29:31.531005 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.530953    4456 scope.go:122] "RemoveContainer" containerID="719ea9ea68dbf199d1ceaca0ef7d177ce497270ff893172917bcc10c42f94fbd"
Nov 02 18:29:31.532407 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.532342    4456 generic.go:356] "Generic (PLEG): container finished" podID="6f751f99-6a9d-42cc-a79f-42980a28c682" containerID="41f175b3acf2c000fe5bfad2439299274523edd1d6fe89f4e991149993716685" exitCode=0
Nov 02 18:29:31.532407 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.532375    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5" event={"ID":"6f751f99-6a9d-42cc-a79f-42980a28c682","Type":"ContainerDied","Data":"41f175b3acf2c000fe5bfad2439299274523edd1d6fe89f4e991149993716685"}
Nov 02 18:29:31.532407 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.532404    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"
Nov 02 18:29:31.532596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.532432    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5" event={"ID":"6f751f99-6a9d-42cc-a79f-42980a28c682","Type":"ContainerDied","Data":"a74b01af13ed5afef21cf4092a9f1518a8de07f0bce4e94ffde0cb9bb2e2fded"}
Nov 02 18:29:31.542727 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.542710    4456 scope.go:122] "RemoveContainer" containerID="719ea9ea68dbf199d1ceaca0ef7d177ce497270ff893172917bcc10c42f94fbd"
Nov 02 18:29:31.543078 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:29:31.543002    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"719ea9ea68dbf199d1ceaca0ef7d177ce497270ff893172917bcc10c42f94fbd\": container with ID starting with 719ea9ea68dbf199d1ceaca0ef7d177ce497270ff893172917bcc10c42f94fbd not found: ID does not exist" containerID="719ea9ea68dbf199d1ceaca0ef7d177ce497270ff893172917bcc10c42f94fbd"
Nov 02 18:29:31.543078 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.543028    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"719ea9ea68dbf199d1ceaca0ef7d177ce497270ff893172917bcc10c42f94fbd"} err="failed to get container status \"719ea9ea68dbf199d1ceaca0ef7d177ce497270ff893172917bcc10c42f94fbd\": rpc error: code = NotFound desc = could not find container \"719ea9ea68dbf199d1ceaca0ef7d177ce497270ff893172917bcc10c42f94fbd\": container with ID starting with 719ea9ea68dbf199d1ceaca0ef7d177ce497270ff893172917bcc10c42f94fbd not found: ID does not exist"
Nov 02 18:29:31.543078 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.543049    4456 scope.go:122] "RemoveContainer" containerID="41f175b3acf2c000fe5bfad2439299274523edd1d6fe89f4e991149993716685"
Nov 02 18:29:31.554566 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.554102    4456 scope.go:122] "RemoveContainer" containerID="41f175b3acf2c000fe5bfad2439299274523edd1d6fe89f4e991149993716685"
Nov 02 18:29:31.555521 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:29:31.555448    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"41f175b3acf2c000fe5bfad2439299274523edd1d6fe89f4e991149993716685\": container with ID starting with 41f175b3acf2c000fe5bfad2439299274523edd1d6fe89f4e991149993716685 not found: ID does not exist" containerID="41f175b3acf2c000fe5bfad2439299274523edd1d6fe89f4e991149993716685"
Nov 02 18:29:31.555521 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.555483    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"41f175b3acf2c000fe5bfad2439299274523edd1d6fe89f4e991149993716685"} err="failed to get container status \"41f175b3acf2c000fe5bfad2439299274523edd1d6fe89f4e991149993716685\": rpc error: code = NotFound desc = could not find container \"41f175b3acf2c000fe5bfad2439299274523edd1d6fe89f4e991149993716685\": container with ID starting with 41f175b3acf2c000fe5bfad2439299274523edd1d6fe89f4e991149993716685 not found: ID does not exist"
Nov 02 18:29:31.561245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.561195    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-v47zv\" (UniqueName: \"kubernetes.io/projected/c9d5829b-2121-4ea6-8d95-e0171152e145-kube-api-access-v47zv\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:29:31.587310 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.587287    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"]
Nov 02 18:29:31.590177 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.590150    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-processing-e2e-ait-ae417f29-57d67f8f77-6wfh5"]
Nov 02 18:29:31.613132 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.613110    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="6f751f99-6a9d-42cc-a79f-42980a28c682" path="/var/lib/kubelet/pods/6f751f99-6a9d-42cc-a79f-42980a28c682/volumes"
Nov 02 18:29:31.625883 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.625862    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"]
Nov 02 18:29:31.629712 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:31.629691    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/payload-pre-processing-e2e-ait-ae417f29-6dd748845f-hx2nc"]
Nov 02 18:29:32.035026 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:32.034992    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:33.331220 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.331162    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:33.612960 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.612882    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="c9d5829b-2121-4ea6-8d95-e0171152e145" path="/var/lib/kubelet/pods/c9d5829b-2121-4ea6-8d95-e0171152e145/volumes"
Nov 02 18:29:33.699583 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.699539    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"]
Nov 02 18:29:33.700149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.700117    4456 cpu_manager.go:414] "containerMap: removing container" podUID="a6e31802-3b39-49c4-980c-13d8ff6dbd79" containerName="maas-api"
Nov 02 18:29:33.700149 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.700141    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="a6e31802-3b39-49c4-980c-13d8ff6dbd79" containerName="maas-api"
Nov 02 18:29:33.700323 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.700174    4456 cpu_manager.go:414] "containerMap: removing container" podUID="c9d5829b-2121-4ea6-8d95-e0171152e145" containerName="payload-pre-processing"
Nov 02 18:29:33.700323 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.700182    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="c9d5829b-2121-4ea6-8d95-e0171152e145" containerName="payload-pre-processing"
Nov 02 18:29:33.700323 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.700219    4456 cpu_manager.go:414] "containerMap: removing container" podUID="6f751f99-6a9d-42cc-a79f-42980a28c682" containerName="payload-processing"
Nov 02 18:29:33.700323 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.700227    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="6f751f99-6a9d-42cc-a79f-42980a28c682" containerName="payload-processing"
Nov 02 18:29:33.700519 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.700340    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="6f751f99-6a9d-42cc-a79f-42980a28c682" containerName="payload-processing"
Nov 02 18:29:33.700519 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.700357    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="a6e31802-3b39-49c4-980c-13d8ff6dbd79" containerName="maas-api"
Nov 02 18:29:33.700519 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.700369    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="c9d5829b-2121-4ea6-8d95-e0171152e145" containerName="payload-pre-processing"
Nov 02 18:29:33.705042 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.705023    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.724094 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.724065    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-ingress\"/\"e2e-authcfg-http-d4ddeb-openshift-default-dockercfg-fhkp5\""
Nov 02 18:29:33.737183 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.737158    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"]
Nov 02 18:29:33.782716 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.782690    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:33.785855 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.785826    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-workload-certs\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.785994 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.785894    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-8hjjx\" (UniqueName: \"kubernetes.io/projected/464f0262-7fb0-4a29-939a-3019ab473a3a-kube-api-access-8hjjx\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.785994 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.785974    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-podinfo\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.786117 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.786028    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-envoy\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.786117 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.786047    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/464f0262-7fb0-4a29-939a-3019ab473a3a-istiod-ca-cert\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.786117 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.786066    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-token\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.786274 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.786126    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-credential-socket\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.786274 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.786177    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-workload-socket\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.786274 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.786257    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-data\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.886984 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.886893    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-workload-certs\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.886984 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.886943    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-8hjjx\" (UniqueName: \"kubernetes.io/projected/464f0262-7fb0-4a29-939a-3019ab473a3a-kube-api-access-8hjjx\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.887215 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.887005    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-podinfo\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.887215 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.887070    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-envoy\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.887215 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.887096    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/464f0262-7fb0-4a29-939a-3019ab473a3a-istiod-ca-cert\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.887215 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.887122    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-token\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.887421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.887155    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-credential-socket\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.887421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.887186    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-workload-socket\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.887421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.887250    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-data\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.887421 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.887364    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-workload-certs\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.887641 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.887578    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-data\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.887777 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.887751    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-credential-socket\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.887892 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.887870    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-workload-socket\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.888060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.888034    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/464f0262-7fb0-4a29-939a-3019ab473a3a-istiod-ca-cert\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.889608 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.889581    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-envoy\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.890079 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.890062    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-podinfo\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.911170 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.911142    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-token\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.911336 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.911313    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-8hjjx\" (UniqueName: \"kubernetes.io/projected/464f0262-7fb0-4a29-939a-3019ab473a3a-kube-api-access-8hjjx\") pod \"e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") " pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:33.941521 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:33.941491    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:34.023035 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:34.022999    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:34.338330 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:34.338291    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"]
Nov 02 18:29:34.338845 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:29:34.338795    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-pod464f0262_7fb0_4a29_939a_3019ab473a3a.slice/crio-9a8099cd4a631c485b1c548893dc34d0145aeb26a9bd7a6e09e60876fac0db3d WatchSource:0}: Error finding container 9a8099cd4a631c485b1c548893dc34d0145aeb26a9bd7a6e09e60876fac0db3d: Status 404 returned error can't find the container with id 9a8099cd4a631c485b1c548893dc34d0145aeb26a9bd7a6e09e60876fac0db3d
Nov 02 18:29:34.340986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:34.340953    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:29:34.341082 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:34.341017    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:29:34.341082 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:34.341045    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:29:34.552641 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:34.552596    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q" event={"ID":"464f0262-7fb0-4a29-939a-3019ab473a3a","Type":"ContainerStarted","Data":"ff795cf460a071b5d906e601c3d3998fa7da1902016e801559ac8258644e9fe6"}
Nov 02 18:29:34.552877 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:34.552670    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q" event={"ID":"464f0262-7fb0-4a29-939a-3019ab473a3a","Type":"ContainerStarted","Data":"9a8099cd4a631c485b1c548893dc34d0145aeb26a9bd7a6e09e60876fac0db3d"}
Nov 02 18:29:34.601262 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:34.601145    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q" podStartSLOduration=1.60112667 podStartE2EDuration="1.60112667s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=false podCreationTimestamp="2026-11-02 18:29:33 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:29:34.599384775 +0000 UTC m=+3591.090447953" watchObservedRunningTime="2026-11-02 18:29:34.60112667 +0000 UTC m=+3591.092189832"
Nov 02 18:29:35.023981 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:35.023944    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:35.029301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:35.029275    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:35.558147 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:35.558119    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:35.559062 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:35.559043    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:37.100366 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:37.100332    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:37.241794 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:37.241762    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:38.804053 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:38.804021    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:38.946297 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:38.946248    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:39.095223 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:39.095143    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:40.136296 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:40.136227    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:40.278653 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:40.278608    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:40.756995 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:40.756965    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:42.557061 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:42.557004    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:44.358469 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:44.358432    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:46.165448 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:29:46.165407    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:29:46.265146 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:46.265115    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:48.536728 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:48.536684    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:49.968458 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:49.968421    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:51.600949 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:51.600909    4456 patch_prober.go:29] interesting pod/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:29:51.601355 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:51.600985    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q" podUID="464f0262-7fb0-4a29-939a-3019ab473a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:29:51.747457 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:51.747419    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:51.990157 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:51.990126    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:52.664849 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:52.664739    4456 generic.go:356] "Generic (PLEG): container finished" podID="464f0262-7fb0-4a29-939a-3019ab473a3a" containerID="ff795cf460a071b5d906e601c3d3998fa7da1902016e801559ac8258644e9fe6" exitCode=0
Nov 02 18:29:52.664849 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:52.664776    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q" event={"ID":"464f0262-7fb0-4a29-939a-3019ab473a3a","Type":"ContainerDied","Data":"ff795cf460a071b5d906e601c3d3998fa7da1902016e801559ac8258644e9fe6"}
Nov 02 18:29:52.665313 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:52.665200    4456 scope.go:122] "RemoveContainer" containerID="ff795cf460a071b5d906e601c3d3998fa7da1902016e801559ac8258644e9fe6"
Nov 02 18:29:52.666592 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:52.666560    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:29:52.666681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:52.666628    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:29:52.666681 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:52.666667    4456 kubelet_resources.go:64] "Allocatable" allocatable={"cpu":"7500m","ephemeral-storage":"114345831029","hugepages-1Gi":"0","hugepages-2Mi":"0","memory":"30892012Ki","pods":"250"}
Nov 02 18:29:53.601410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:53.601369    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:53.672243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:53.672206    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q" event={"ID":"464f0262-7fb0-4a29-939a-3019ab473a3a","Type":"ContainerStarted","Data":"3cac2756465226719282b5e1ccdab4dc8d0ed7ad58176349539ec3d0d1fbf810"}
Nov 02 18:29:54.023847 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:54.023792    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="unhealthy" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:54.024338 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:54.024029    4456 patch_prober.go:29] interesting pod/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Nov 02 18:29:54.024338 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:54.024092    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q" podUID="464f0262-7fb0-4a29-939a-3019ab473a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Nov 02 18:29:55.023937 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:55.023901    4456 patch_prober.go:29] interesting pod/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Nov 02 18:29:55.024364 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:55.023979    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q" podUID="464f0262-7fb0-4a29-939a-3019ab473a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Nov 02 18:29:55.368880 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:55.368773    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:56.023660 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:56.023623    4456 patch_prober.go:29] interesting pod/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Nov 02 18:29:56.023859 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:56.023687    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q" podUID="464f0262-7fb0-4a29-939a-3019ab473a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Nov 02 18:29:57.023790 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:57.023749    4456 patch_prober.go:29] interesting pod/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Nov 02 18:29:57.024211 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:57.023830    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q" podUID="464f0262-7fb0-4a29-939a-3019ab473a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Nov 02 18:29:57.268214 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:57.268177    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:29:58.023792 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:58.023758    4456 patch_prober.go:29] interesting pod/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q container/istio-proxy namespace/openshift-ingress: Startup probe status=failure output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused" start-of-body=
Nov 02 18:29:58.024213 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:58.023839    4456 prober.go:122] "Probe failed" probeType="Startup" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q" podUID="464f0262-7fb0-4a29-939a-3019ab473a3a" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.59:15021/healthz/ready\": dial tcp 10.134.1.59:15021: connect: connection refused"
Nov 02 18:29:59.027972 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:59.027942    4456 kubelet.go:2727] "SyncLoop (probe)" probe="startup" status="started" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:59.028438 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:59.028421    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:59.029058 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:59.029038    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:29:59.050302 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:29:59.050279    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:00.156932 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:00.156899    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"]
Nov 02 18:30:00.159763 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:00.159739    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"
Nov 02 18:30:00.183530 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:00.183503    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"]
Nov 02 18:30:00.190356 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:00.190331    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-94lnf\" (UniqueName: \"kubernetes.io/projected/d4e4289f-eff9-4caa-9b01-7e0bc629a540-kube-api-access-94lnf\") pod \"maas-api-key-cleanup-29894070-q69np\" (UID: \"d4e4289f-eff9-4caa-9b01-7e0bc629a540\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"
Nov 02 18:30:00.190533 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:00.190514    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e4289f-eff9-4caa-9b01-7e0bc629a540-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894070-q69np\" (UID: \"d4e4289f-eff9-4caa-9b01-7e0bc629a540\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"
Nov 02 18:30:00.274311 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:00.274283    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"odh-ai-gateway-infra\"/\"maas-api-cleanup-dockercfg-x9vwr\""
Nov 02 18:30:00.291531 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:00.291501    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e4289f-eff9-4caa-9b01-7e0bc629a540-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894070-q69np\" (UID: \"d4e4289f-eff9-4caa-9b01-7e0bc629a540\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"
Nov 02 18:30:00.291688 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:00.291557    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-94lnf\" (UniqueName: \"kubernetes.io/projected/d4e4289f-eff9-4caa-9b01-7e0bc629a540-kube-api-access-94lnf\") pod \"maas-api-key-cleanup-29894070-q69np\" (UID: \"d4e4289f-eff9-4caa-9b01-7e0bc629a540\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"
Nov 02 18:30:00.292130 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:00.292112    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e4289f-eff9-4caa-9b01-7e0bc629a540-maas-api-service-ca\") pod \"maas-api-key-cleanup-29894070-q69np\" (UID: \"d4e4289f-eff9-4caa-9b01-7e0bc629a540\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"
Nov 02 18:30:00.316982 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:00.316953    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-94lnf\" (UniqueName: \"kubernetes.io/projected/d4e4289f-eff9-4caa-9b01-7e0bc629a540-kube-api-access-94lnf\") pod \"maas-api-key-cleanup-29894070-q69np\" (UID: \"d4e4289f-eff9-4caa-9b01-7e0bc629a540\") " pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"
Nov 02 18:30:00.472738 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:00.472711    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"
Nov 02 18:30:00.770420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:00.770379    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"]
Nov 02 18:30:00.771152 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:30:00.771122    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-burstable.slice/kubepods-burstable-podd4e4289f_eff9_4caa_9b01_7e0bc629a540.slice/crio-592cde87131a939a4839b85d15cd45ed4eedb81a2506279afb4598c69dd5b28c WatchSource:0}: Error finding container 592cde87131a939a4839b85d15cd45ed4eedb81a2506279afb4598c69dd5b28c: Status 404 returned error can't find the container with id 592cde87131a939a4839b85d15cd45ed4eedb81a2506279afb4598c69dd5b28c
Nov 02 18:30:00.861033 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:00.861004    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:01.739155 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:01.739126    4456 generic.go:356] "Generic (PLEG): container finished" podID="d4e4289f-eff9-4caa-9b01-7e0bc629a540" containerID="0282f35b9616a6c952125799e6dc5718ebbc380b3c6ccdd3410485d0dd98adf3" exitCode=60
Nov 02 18:30:01.739575 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:01.739217    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np" event={"ID":"d4e4289f-eff9-4caa-9b01-7e0bc629a540","Type":"ContainerDied","Data":"0282f35b9616a6c952125799e6dc5718ebbc380b3c6ccdd3410485d0dd98adf3"}
Nov 02 18:30:01.739575 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:01.739276    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np" event={"ID":"d4e4289f-eff9-4caa-9b01-7e0bc629a540","Type":"ContainerStarted","Data":"592cde87131a939a4839b85d15cd45ed4eedb81a2506279afb4598c69dd5b28c"}
Nov 02 18:30:01.739575 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:01.739448    4456 scope.go:122] "RemoveContainer" containerID="0282f35b9616a6c952125799e6dc5718ebbc380b3c6ccdd3410485d0dd98adf3"
Nov 02 18:30:02.746703 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:02.746617    4456 generic.go:356] "Generic (PLEG): container finished" podID="d4e4289f-eff9-4caa-9b01-7e0bc629a540" containerID="4f751baeddd962c77048f39e6901cdd51c8356088c002334d8c730eca66dc30f" exitCode=60
Nov 02 18:30:02.746703 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:02.746673    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np" event={"ID":"d4e4289f-eff9-4caa-9b01-7e0bc629a540","Type":"ContainerDied","Data":"4f751baeddd962c77048f39e6901cdd51c8356088c002334d8c730eca66dc30f"}
Nov 02 18:30:02.748647 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:02.746724    4456 scope.go:122] "RemoveContainer" containerID="0282f35b9616a6c952125799e6dc5718ebbc380b3c6ccdd3410485d0dd98adf3"
Nov 02 18:30:02.748647 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:02.746994    4456 scope.go:122] "RemoveContainer" containerID="4f751baeddd962c77048f39e6901cdd51c8356088c002334d8c730eca66dc30f"
Nov 02 18:30:02.748647 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:30:02.747226    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894070-q69np_odh-ai-gateway-infra(d4e4289f-eff9-4caa-9b01-7e0bc629a540)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np" podUID="d4e4289f-eff9-4caa-9b01-7e0bc629a540"
Nov 02 18:30:02.790947 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:02.790914    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:03.754099 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:03.754069    4456 scope.go:122] "RemoveContainer" containerID="4f751baeddd962c77048f39e6901cdd51c8356088c002334d8c730eca66dc30f"
Nov 02 18:30:03.754503 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:30:03.754292    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 10s restarting failed container=cleanup pod=maas-api-key-cleanup-29894070-q69np_odh-ai-gateway-infra(d4e4289f-eff9-4caa-9b01-7e0bc629a540)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np" podUID="d4e4289f-eff9-4caa-9b01-7e0bc629a540"
Nov 02 18:30:04.547226 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:04.547188    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:06.344516 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:06.344421    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:08.147971 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:08.147938    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:09.948604 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:09.948573    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:11.752633 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:11.752596    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:13.549895 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:13.549859    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:13.611337 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:13.611311    4456 scope.go:122] "RemoveContainer" containerID="4f751baeddd962c77048f39e6901cdd51c8356088c002334d8c730eca66dc30f"
Nov 02 18:30:14.392597 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:14.392571    4456 scope.go:122] "RemoveContainer" containerID="240a29cc3ce6c00cbe75b862d47d92e5e891e23abaa4fa7ea261b20767c9569c"
Nov 02 18:30:14.813655 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:14.813626    4456 generic.go:356] "Generic (PLEG): container finished" podID="d4e4289f-eff9-4caa-9b01-7e0bc629a540" containerID="4720bcf00f0b9f389ed4bdb090e655246d260c8833bf0e782ebc1f9f4cfdee1f" exitCode=60
Nov 02 18:30:14.814167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:14.813705    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np" event={"ID":"d4e4289f-eff9-4caa-9b01-7e0bc629a540","Type":"ContainerDied","Data":"4720bcf00f0b9f389ed4bdb090e655246d260c8833bf0e782ebc1f9f4cfdee1f"}
Nov 02 18:30:14.814167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:14.813770    4456 scope.go:122] "RemoveContainer" containerID="4f751baeddd962c77048f39e6901cdd51c8356088c002334d8c730eca66dc30f"
Nov 02 18:30:14.814167 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:14.814154    4456 scope.go:122] "RemoveContainer" containerID="4720bcf00f0b9f389ed4bdb090e655246d260c8833bf0e782ebc1f9f4cfdee1f"
Nov 02 18:30:14.814479 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:30:14.814458    4456 pod_workers.go:1324] "Error syncing pod, skipping" err="failed to \"StartContainer\" for \"cleanup\" with CrashLoopBackOff: \"back-off 20s restarting failed container=cleanup pod=maas-api-key-cleanup-29894070-q69np_odh-ai-gateway-infra(d4e4289f-eff9-4caa-9b01-7e0bc629a540)\"" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np" podUID="d4e4289f-eff9-4caa-9b01-7e0bc629a540"
Nov 02 18:30:15.377440 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:15.377364    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:15.869387 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:15.869353    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"]
Nov 02 18:30:16.012373 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:16.012345    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"
Nov 02 18:30:16.150646 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:16.150559    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/d4e4289f-eff9-4caa-9b01-7e0bc629a540-kube-api-access-94lnf\" (UniqueName: \"kubernetes.io/projected/d4e4289f-eff9-4caa-9b01-7e0bc629a540-kube-api-access-94lnf\") pod \"d4e4289f-eff9-4caa-9b01-7e0bc629a540\" (UID: \"d4e4289f-eff9-4caa-9b01-7e0bc629a540\") "
Nov 02 18:30:16.150646 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:16.150635    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/d4e4289f-eff9-4caa-9b01-7e0bc629a540-maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e4289f-eff9-4caa-9b01-7e0bc629a540-maas-api-service-ca\") pod \"d4e4289f-eff9-4caa-9b01-7e0bc629a540\" (UID: \"d4e4289f-eff9-4caa-9b01-7e0bc629a540\") "
Nov 02 18:30:16.151022 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:16.151000    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/d4e4289f-eff9-4caa-9b01-7e0bc629a540-maas-api-service-ca" pod "d4e4289f-eff9-4caa-9b01-7e0bc629a540" (UID: "d4e4289f-eff9-4caa-9b01-7e0bc629a540"). InnerVolumeSpecName "maas-api-service-ca". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:30:16.152729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:16.152706    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/d4e4289f-eff9-4caa-9b01-7e0bc629a540-kube-api-access-94lnf" pod "d4e4289f-eff9-4caa-9b01-7e0bc629a540" (UID: "d4e4289f-eff9-4caa-9b01-7e0bc629a540"). InnerVolumeSpecName "kube-api-access-94lnf". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:30:16.165442 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:30:16.165413    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:30:16.251576 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:16.251541    4456 reconciler_common.go:299] "Volume detached for volume \"maas-api-service-ca\" (UniqueName: \"kubernetes.io/configmap/d4e4289f-eff9-4caa-9b01-7e0bc629a540-maas-api-service-ca\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:30:16.251576 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:16.251564    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-94lnf\" (UniqueName: \"kubernetes.io/projected/d4e4289f-eff9-4caa-9b01-7e0bc629a540-kube-api-access-94lnf\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:30:16.826209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:16.826176    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"
Nov 02 18:30:16.826209 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:16.826205    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np" event={"ID":"d4e4289f-eff9-4caa-9b01-7e0bc629a540","Type":"ContainerDied","Data":"592cde87131a939a4839b85d15cd45ed4eedb81a2506279afb4598c69dd5b28c"}
Nov 02 18:30:16.826479 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:16.826260    4456 scope.go:122] "RemoveContainer" containerID="4720bcf00f0b9f389ed4bdb090e655246d260c8833bf0e782ebc1f9f4cfdee1f"
Nov 02 18:30:16.881243 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:16.881219    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"]
Nov 02 18:30:16.894420 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:16.894394    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-key-cleanup-29894070-q69np"]
Nov 02 18:30:17.152930 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:17.152851    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:17.612444 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:17.612413    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="d4e4289f-eff9-4caa-9b01-7e0bc629a540" path="/var/lib/kubelet/pods/d4e4289f-eff9-4caa-9b01-7e0bc629a540/volumes"
Nov 02 18:30:18.871959 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:18.871925    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:19.035914 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:19.035878    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:19.175888 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:19.175792    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:20.876131 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:20.876092    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:22.030499 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:22.030462    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"]
Nov 02 18:30:22.043547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:22.043521    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d576e68b-rtv5w"]
Nov 02 18:30:22.654385 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:22.654340    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:23.613274 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:23.613243    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="13f6ef0d-fbff-49c2-b187-d4340496ee8b" path="/var/lib/kubelet/pods/13f6ef0d-fbff-49c2-b187-d4340496ee8b/volumes"
Nov 02 18:30:24.459406 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:24.459358    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:26.252182 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:26.252137    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:28.007516 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:28.007480    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:28.151552 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:28.151516    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:29.853329 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:29.853282    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:31.645843 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:31.645790    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:33.456808 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:33.456775    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:35.249317 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:35.249269    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:37.163090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:37.163051    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:39.009534 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:39.009499    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:40.255039 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:40.255005    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:40.770882 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:40.770844    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:42.561666 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:42.561628    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:44.030036 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:44.030001    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"]
Nov 02 18:30:44.042664 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:44.042630    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-d2bc93bc-v6ccg"]
Nov 02 18:30:44.361102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:44.360998    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:45.613090 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:45.613057    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="91ff46e3-3e04-4708-ade1-9097b6793298" path="/var/lib/kubelet/pods/91ff46e3-3e04-4708-ade1-9097b6793298/volumes"
Nov 02 18:30:46.149025 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:46.148986    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:46.170181 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:30:46.170143    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:30:47.946693 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:47.946658    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:49.743959 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:49.743897    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:51.558271 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:51.558232    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:53.350966 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:53.350933    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:55.290624 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:55.290583    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:57.068399 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:57.068368    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:30:58.953349 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:30:58.953317    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:00.760078 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:00.759872    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:02.555070 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:02.555006    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:04.443900 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:04.443797    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:06.268301 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:06.268219    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:07.950430 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:07.950398    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:08.094785 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:08.094755    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:09.856714 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:09.856679    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:11.391425 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:11.391399    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-driver/0.log"
Nov 02 18:31:11.437689 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:11.437666    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-cluster-csi-drivers_aws-ebs-csi-driver-node-6bb2n_758467e6-69e4-4c1f-b344-e1e74497c298/csi-driver/0.log"
Nov 02 18:31:11.660345 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:11.660227    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:13.572408 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:13.572373    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:15.475160 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:15.475129    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:16.164780 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:31:16.164742    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:31:17.349729 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:17.349690    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:19.165724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:19.165676    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:21.064122 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:21.064089    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:22.868043 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:22.868012    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:23.007041 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:23.007003    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:29.486368 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:29.486333    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:30.033853 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:30.033798    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:30.175011 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:30.174981    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:30.764250 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:30.764217    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:37.030746 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:37.030712    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:37.177282 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:37.177251    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:37.312440 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:37.312354    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"]
Nov 02 18:31:37.312620 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:37.312590    4456 kuberuntime_container.go:898] "Killing container with a grace period" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q" podUID="464f0262-7fb0-4a29-939a-3019ab473a3a" containerName="istio-proxy" containerID="cri-o://3cac2756465226719282b5e1ccdab4dc8d0ed7ad58176349539ec3d0d1fbf810" gracePeriod=30
Nov 02 18:31:37.386394 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:37.386307    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:40.412715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:40.412662    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:31:42.780874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.777918    4456 patch_prober.go:29] interesting pod/maas-default-gateway-openshift-default-76f4876667-lrssd container/istio-proxy namespace/openshift-ingress: Readiness probe status=failure output="Get \"http://10.134.1.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)" start-of-body=
Nov 02 18:31:42.780874 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.778007    4456 prober.go:122] "Probe failed" probeType="Readiness" pod="openshift-ingress/maas-default-gateway-openshift-default-76f4876667-lrssd" podUID="9d114747-c309-445d-8159-5e532bede175" containerName="istio-proxy" probeResult="failure" output="Get \"http://10.134.1.27:15021/healthz/ready\": context deadline exceeded (Client.Timeout exceeded while awaiting headers)"
Nov 02 18:31:42.946626 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.946577    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:31:42.978660 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.978618    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/downward-api/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-podinfo\") pod \"464f0262-7fb0-4a29-939a-3019ab473a3a\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") "
Nov 02 18:31:42.978852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.978705    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-credential-socket\") pod \"464f0262-7fb0-4a29-939a-3019ab473a3a\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") "
Nov 02 18:31:42.978852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.978756    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-data\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-data\") pod \"464f0262-7fb0-4a29-939a-3019ab473a3a\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") "
Nov 02 18:31:42.978986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.978864    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-workload-socket\") pod \"464f0262-7fb0-4a29-939a-3019ab473a3a\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") "
Nov 02 18:31:42.978986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.978899    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-workload-certs\") pod \"464f0262-7fb0-4a29-939a-3019ab473a3a\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") "
Nov 02 18:31:42.978986 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.978930    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-token\" (UniqueName: \"kubernetes.io/projected/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-token\") pod \"464f0262-7fb0-4a29-939a-3019ab473a3a\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") "
Nov 02 18:31:42.979154 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.978993    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/projected/464f0262-7fb0-4a29-939a-3019ab473a3a-kube-api-access-8hjjx\" (UniqueName: \"kubernetes.io/projected/464f0262-7fb0-4a29-939a-3019ab473a3a-kube-api-access-8hjjx\") pod \"464f0262-7fb0-4a29-939a-3019ab473a3a\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") "
Nov 02 18:31:42.979154 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.979021    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/configmap/464f0262-7fb0-4a29-939a-3019ab473a3a-istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/464f0262-7fb0-4a29-939a-3019ab473a3a-istiod-ca-cert\") pod \"464f0262-7fb0-4a29-939a-3019ab473a3a\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") "
Nov 02 18:31:42.979154 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.979087    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-data" pod "464f0262-7fb0-4a29-939a-3019ab473a3a" (UID: "464f0262-7fb0-4a29-939a-3019ab473a3a"). InnerVolumeSpecName "istio-data". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:42.979305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.979184    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-workload-certs" pod "464f0262-7fb0-4a29-939a-3019ab473a3a" (UID: "464f0262-7fb0-4a29-939a-3019ab473a3a"). InnerVolumeSpecName "workload-certs". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:42.979305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.979271    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-workload-socket" pod "464f0262-7fb0-4a29-939a-3019ab473a3a" (UID: "464f0262-7fb0-4a29-939a-3019ab473a3a"). InnerVolumeSpecName "workload-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:42.979526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.979409    4456 reconciler_common.go:299] "Volume detached for volume \"workload-socket\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-workload-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:31:42.979526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.979510    4456 reconciler_common.go:299] "Volume detached for volume \"workload-certs\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-workload-certs\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:31:42.979526 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.979525    4456 reconciler_common.go:299] "Volume detached for volume \"istio-data\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-data\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:31:42.979892 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.979865    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-credential-socket" pod "464f0262-7fb0-4a29-939a-3019ab473a3a" (UID: "464f0262-7fb0-4a29-939a-3019ab473a3a"). InnerVolumeSpecName "credential-socket". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:42.981794 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.981768    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/downward-api/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-podinfo" pod "464f0262-7fb0-4a29-939a-3019ab473a3a" (UID: "464f0262-7fb0-4a29-939a-3019ab473a3a"). InnerVolumeSpecName "istio-podinfo". PluginName "kubernetes.io/downward-api", VolumeGIDValue ""
Nov 02 18:31:42.982053 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.982017    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464f0262-7fb0-4a29-939a-3019ab473a3a-kube-api-access-8hjjx" pod "464f0262-7fb0-4a29-939a-3019ab473a3a" (UID: "464f0262-7fb0-4a29-939a-3019ab473a3a"). InnerVolumeSpecName "kube-api-access-8hjjx". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:31:42.982137 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.982098    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/projected/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-token" pod "464f0262-7fb0-4a29-939a-3019ab473a3a" (UID: "464f0262-7fb0-4a29-939a-3019ab473a3a"). InnerVolumeSpecName "istio-token". PluginName "kubernetes.io/projected", VolumeGIDValue ""
Nov 02 18:31:42.982950 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:42.982920    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/configmap/464f0262-7fb0-4a29-939a-3019ab473a3a-istiod-ca-cert" pod "464f0262-7fb0-4a29-939a-3019ab473a3a" (UID: "464f0262-7fb0-4a29-939a-3019ab473a3a"). InnerVolumeSpecName "istiod-ca-cert". PluginName "kubernetes.io/configmap", VolumeGIDValue ""
Nov 02 18:31:43.080483 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.080454    4456 reconciler_common.go:163] "operationExecutor.UnmountVolume started for volume \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-envoy\") pod \"464f0262-7fb0-4a29-939a-3019ab473a3a\" (UID: \"464f0262-7fb0-4a29-939a-3019ab473a3a\") "
Nov 02 18:31:43.080667 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.080655    4456 reconciler_common.go:299] "Volume detached for volume \"istio-token\" (UniqueName: \"kubernetes.io/projected/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-token\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:31:43.080715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.080670    4456 reconciler_common.go:299] "Volume detached for volume \"kube-api-access-8hjjx\" (UniqueName: \"kubernetes.io/projected/464f0262-7fb0-4a29-939a-3019ab473a3a-kube-api-access-8hjjx\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:31:43.080715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.080679    4456 reconciler_common.go:299] "Volume detached for volume \"istiod-ca-cert\" (UniqueName: \"kubernetes.io/configmap/464f0262-7fb0-4a29-939a-3019ab473a3a-istiod-ca-cert\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:31:43.080715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.080687    4456 reconciler_common.go:299] "Volume detached for volume \"istio-podinfo\" (UniqueName: \"kubernetes.io/downward-api/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-podinfo\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:31:43.080715 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.080695    4456 reconciler_common.go:299] "Volume detached for volume \"credential-socket\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-credential-socket\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:31:43.082623 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.082598    4456 operation_generator.go:780] UnmountVolume.TearDown succeeded for volume "kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-envoy" pod "464f0262-7fb0-4a29-939a-3019ab473a3a" (UID: "464f0262-7fb0-4a29-939a-3019ab473a3a"). InnerVolumeSpecName "istio-envoy". PluginName "kubernetes.io/empty-dir", VolumeGIDValue ""
Nov 02 18:31:43.181400 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.181313    4456 reconciler_common.go:299] "Volume detached for volume \"istio-envoy\" (UniqueName: \"kubernetes.io/empty-dir/464f0262-7fb0-4a29-939a-3019ab473a3a-istio-envoy\") on node \"ip-10-0-141-87.ec2.internal\" DevicePath \"\""
Nov 02 18:31:43.274558 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.274523    4456 generic.go:356] "Generic (PLEG): container finished" podID="464f0262-7fb0-4a29-939a-3019ab473a3a" containerID="3cac2756465226719282b5e1ccdab4dc8d0ed7ad58176349539ec3d0d1fbf810" exitCode=0
Nov 02 18:31:43.274734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.274576    4456 util.go:52] "No ready sandbox for pod can be found. Need to start a new one" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"
Nov 02 18:31:43.274734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.274619    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q" event={"ID":"464f0262-7fb0-4a29-939a-3019ab473a3a","Type":"ContainerDied","Data":"3cac2756465226719282b5e1ccdab4dc8d0ed7ad58176349539ec3d0d1fbf810"}
Nov 02 18:31:43.274734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.274669    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q" event={"ID":"464f0262-7fb0-4a29-939a-3019ab473a3a","Type":"ContainerDied","Data":"9a8099cd4a631c485b1c548893dc34d0145aeb26a9bd7a6e09e60876fac0db3d"}
Nov 02 18:31:43.274734 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.274694    4456 scope.go:122] "RemoveContainer" containerID="3cac2756465226719282b5e1ccdab4dc8d0ed7ad58176349539ec3d0d1fbf810"
Nov 02 18:31:43.284048 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.284028    4456 scope.go:122] "RemoveContainer" containerID="ff795cf460a071b5d906e601c3d3998fa7da1902016e801559ac8258644e9fe6"
Nov 02 18:31:43.292676 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.292660    4456 scope.go:122] "RemoveContainer" containerID="3cac2756465226719282b5e1ccdab4dc8d0ed7ad58176349539ec3d0d1fbf810"
Nov 02 18:31:43.292923 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:31:43.292906    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"3cac2756465226719282b5e1ccdab4dc8d0ed7ad58176349539ec3d0d1fbf810\": container with ID starting with 3cac2756465226719282b5e1ccdab4dc8d0ed7ad58176349539ec3d0d1fbf810 not found: ID does not exist" containerID="3cac2756465226719282b5e1ccdab4dc8d0ed7ad58176349539ec3d0d1fbf810"
Nov 02 18:31:43.292995 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.292931    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"3cac2756465226719282b5e1ccdab4dc8d0ed7ad58176349539ec3d0d1fbf810"} err="failed to get container status \"3cac2756465226719282b5e1ccdab4dc8d0ed7ad58176349539ec3d0d1fbf810\": rpc error: code = NotFound desc = could not find container \"3cac2756465226719282b5e1ccdab4dc8d0ed7ad58176349539ec3d0d1fbf810\": container with ID starting with 3cac2756465226719282b5e1ccdab4dc8d0ed7ad58176349539ec3d0d1fbf810 not found: ID does not exist"
Nov 02 18:31:43.292995 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.292949    4456 scope.go:122] "RemoveContainer" containerID="ff795cf460a071b5d906e601c3d3998fa7da1902016e801559ac8258644e9fe6"
Nov 02 18:31:43.293195 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:31:43.293177    4456 log.go:32] "ContainerStatus from runtime service failed" err="rpc error: code = NotFound desc = could not find container \"ff795cf460a071b5d906e601c3d3998fa7da1902016e801559ac8258644e9fe6\": container with ID starting with ff795cf460a071b5d906e601c3d3998fa7da1902016e801559ac8258644e9fe6 not found: ID does not exist" containerID="ff795cf460a071b5d906e601c3d3998fa7da1902016e801559ac8258644e9fe6"
Nov 02 18:31:43.293236 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.293198    4456 pod_container_deletor.go:53] "DeleteContainer returned error" containerID={"Type":"cri-o","ID":"ff795cf460a071b5d906e601c3d3998fa7da1902016e801559ac8258644e9fe6"} err="failed to get container status \"ff795cf460a071b5d906e601c3d3998fa7da1902016e801559ac8258644e9fe6\": rpc error: code = NotFound desc = could not find container \"ff795cf460a071b5d906e601c3d3998fa7da1902016e801559ac8258644e9fe6\": container with ID starting with ff795cf460a071b5d906e601c3d3998fa7da1902016e801559ac8258644e9fe6 not found: ID does not exist"
Nov 02 18:31:43.331604 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.331577    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"]
Nov 02 18:31:43.345078 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.345052    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["openshift-ingress/e2e-authcfg-http-d4ddeb-openshift-default-57468bc5fc-br58q"]
Nov 02 18:31:43.612913 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:31:43.612882    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="464f0262-7fb0-4a29-939a-3019ab473a3a" path="/var/lib/kubelet/pods/464f0262-7fb0-4a29-939a-3019ab473a3a/volumes"
Nov 02 18:31:46.174716 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:31:46.174672    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:32:14.437875 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:14.437791    4456 scope.go:122] "RemoveContainer" containerID="d793c2237d65c37419b1d637f39a881b7ba53dbb7ddc89c82d94a62d448439ec"
Nov 02 18:32:14.446531 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:14.446510    4456 scope.go:122] "RemoveContainer" containerID="011a1cee76f493227a0d5cb4603055f4bcc209fc89702118841db690ca2067fc"
Nov 02 18:32:16.165143 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:32:16.165105    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:32:34.870451 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:34.870420    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5b7cd5c7-brdvs_040ec9bf-319e-4bae-b233-4e4e7e74a379/authorino/0.log"
Nov 02 18:32:40.549290 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:40.549248    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:32:46.167504 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:32:46.167459    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:32:48.354239 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:48.354188    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-64678b9774-s75tm_522b0328-e5fe-4a99-8c63-39635d795116/maas-api/0.log"
Nov 02 18:32:48.786893 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:48.786846    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-s82hw_9548e1c9-9efe-4459-ac94-ae198fb8fe99/revoke-keys/0.log"
Nov 02 18:32:48.975138 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:48.975112    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-452b6d7b-p9mpc_8b1a1bfa-b8ef-41d7-9e02-81817b8c2883/revoke-keys/0.log"
Nov 02 18:32:49.171265 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:49.171183    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-53e082c1-wscp6_67106641-ccc8-4c65-9069-d69049611fa3/revoke-keys/0.log"
Nov 02 18:32:51.250160 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:51.250128    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5b7cd5c7-brdvs_040ec9bf-319e-4bae-b233-4e4e7e74a379/authorino/0.log"
Nov 02 18:32:51.889558 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:51.889529    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-c8vr7_08c9aa32-dadc-4112-b608-4cc64f56295c/kuadrant-console-plugin/0.log"
Nov 02 18:32:52.114766 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:52.114737    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-47dcf_39b9a295-67cb-402a-9f11-41d0e8e7d2eb/manager/0.log"
Nov 02 18:32:53.958599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:53.958570    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78648788bf-npsk7_347c8613-0791-494e-83e3-faded5593d88/kube-auth-proxy/0.log"
Nov 02 18:32:54.558669 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:54.558638    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-5knvx_8ef79a10-292b-42d9-aab0-53bee3e43ed2/istio-proxy/0.log"
Nov 02 18:32:55.554112 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:55.554082    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-lrssd_9d114747-c309-445d-8159-5e532bede175/istio-proxy/0.log"
Nov 02 18:32:55.764463 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:55.764433    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-r5jnj_cce08d5b-d93c-4c5f-bb03-89c460372a06/istio-proxy/0.log"
Nov 02 18:32:57.030881 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:57.030847    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"]
Nov 02 18:32:57.043852 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:57.043801    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-452b6d7b-p9mpc"]
Nov 02 18:32:57.562024 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:57.561995    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4_6e44698a-838e-4eea-b956-585673963ed9/main/0.log"
Nov 02 18:32:57.612016 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:57.611984    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="8b1a1bfa-b8ef-41d7-9e02-81817b8c2883" path="/var/lib/kubelet/pods/8b1a1bfa-b8ef-41d7-9e02-81817b8c2883/volumes"
Nov 02 18:32:58.166300 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:58.166269    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-xc9kg_3b0e614e-16b1-4e28-bea6-4b7b9a62b177/main/0.log"
Nov 02 18:32:58.752459 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:58.752429    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k_14d43510-cfbb-4ddc-88cb-ccd4cb3a543d/main/0.log"
Nov 02 18:32:59.848911 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:32:59.848877    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5b7cd5c7-brdvs_040ec9bf-319e-4bae-b233-4e4e7e74a379/authorino/0.log"
Nov 02 18:33:11.375521 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:11.375490    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-64678b9774-s75tm_522b0328-e5fe-4a99-8c63-39635d795116/maas-api/0.log"
Nov 02 18:33:11.724030 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:11.723991    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:11.781004 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:11.780966    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-3e31a506-s82hw_9548e1c9-9efe-4459-ac94-ae198fb8fe99/revoke-keys/0.log"
Nov 02 18:33:11.862102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:11.862055    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:11.973469 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:11.973442    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/odh-ai-gateway-infra_maas-api-revoke-sub-53e082c1-wscp6_67106641-ccc8-4c65-9069-d69049611fa3/revoke-keys/0.log"
Nov 02 18:33:13.694865 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:13.694809    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5b7cd5c7-brdvs_040ec9bf-319e-4bae-b233-4e4e7e74a379/authorino/0.log"
Nov 02 18:33:14.318119 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:14.318091    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-c8vr7_08c9aa32-dadc-4112-b608-4cc64f56295c/kuadrant-console-plugin/0.log"
Nov 02 18:33:14.510724 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:14.510698    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-47dcf_39b9a295-67cb-402a-9f11-41d0e8e7d2eb/manager/0.log"
Nov 02 18:33:16.093756 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:16.093730    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78648788bf-npsk7_347c8613-0791-494e-83e3-faded5593d88/kube-auth-proxy/0.log"
Nov 02 18:33:16.164680 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:33:16.164645    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:33:16.697546 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:16.697512    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-5knvx_8ef79a10-292b-42d9-aab0-53bee3e43ed2/istio-proxy/0.log"
Nov 02 18:33:17.672718 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:17.672683    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-lrssd_9d114747-c309-445d-8159-5e532bede175/istio-proxy/0.log"
Nov 02 18:33:17.883655 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:17.883621    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-r5jnj_cce08d5b-d93c-4c5f-bb03-89c460372a06/istio-proxy/0.log"
Nov 02 18:33:18.031830 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:18.031771    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"]
Nov 02 18:33:18.044027 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:18.043999    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-3e31a506-s82hw"]
Nov 02 18:33:19.372993 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:19.372964    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-distinct-2-simulated-kserve-cbd4f5586-8rrp4_6e44698a-838e-4eea-b956-585673963ed9/main/0.log"
Nov 02 18:33:19.612869 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:19.612833    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="9548e1c9-9efe-4459-ac94-ae198fb8fe99" path="/var/lib/kubelet/pods/9548e1c9-9efe-4459-ac94-ae198fb8fe99/volumes"
Nov 02 18:33:19.944652 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:19.944613    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_e2e-trlp-test-simulated-kserve-69b588897d-xc9kg_3b0e614e-16b1-4e28-bea6-4b7b9a62b177/main/0.log"
Nov 02 18:33:20.532245 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:20.532213    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/llm_facebook-opt-125m-simulated-kserve-84b4ff656c-v7b9k_14d43510-cfbb-4ddc-88cb-ccd4cb3a543d/main/0.log"
Nov 02 18:33:35.504666 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:35.504632    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:40.629548 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:40.629520    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:33:41.040128 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:41.040060    4456 kubelet.go:2624] "SyncLoop DELETE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"]
Nov 02 18:33:41.051789 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:41.051755    4456 kubelet.go:2618] "SyncLoop REMOVE" source="api" pods=["odh-ai-gateway-infra/maas-api-revoke-sub-53e082c1-wscp6"]
Nov 02 18:33:41.612431 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:33:41.612385    4456 kubelet_volumes.go:161] "Cleaned up orphaned pod volumes dir" podUID="67106641-ccc8-4c65-9069-d69049611fa3" path="/var/lib/kubelet/pods/67106641-ccc8-4c65-9069-d69049611fa3/volumes"
Nov 02 18:33:46.164421 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:33:46.164382    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:34:14.480197 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:14.480165    4456 scope.go:122] "RemoveContainer" containerID="96aef0e60dace19aff710f43029bace1c4404ccb5a8dd28db8141443c21ac8f0"
Nov 02 18:34:14.489598 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:14.489576    4456 scope.go:122] "RemoveContainer" containerID="41e22f6f9c879beca7ade73ce0e544b60e858a77e4ce565e74be6402327b8cd1"
Nov 02 18:34:15.592570 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:15.592539    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:16.165139 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:34:16.165107    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:34:33.095730 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:33.095683    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:40.018302 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:40.018269    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:40.240596 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:40.240559    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:40.743456 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:40.743415    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:34:45.074558 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:45.074522    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_global-pull-secret-syncer-mmhd7_8fe19c0f-fd25-49b6-bb3a-9ad60c173c9d/global-pull-secret-syncer/0.log"
Nov 02 18:34:45.910930 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:45.910896    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log"
Nov 02 18:34:46.164692 ip-10-0-141-87 kubenswrapper[4456]: E1102 18:34:46.164604    4456 prober.go:265] "Unable to write all bytes from execInContainer" err="short write" expectedBytes=11141 actualBytes=10240 pod="openshift-insights/insights-runtime-extractor-6bfcf" containerName="extractor"
Nov 02 18:34:46.554391 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:46.554365    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kube-system_kube-apiserver-proxy-ip-10-0-141-87.ec2.internal_fbfcd267e5bdda32002f5b72a55d4954/haproxy/0.log"
Nov 02 18:34:52.581685 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:52.581645    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_authorino-6f5b7cd5c7-brdvs_040ec9bf-319e-4bae-b233-4e4e7e74a379/authorino/0.log"
Nov 02 18:34:52.947806 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:52.947778    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-console-plugin-66c4fb575d-c8vr7_08c9aa32-dadc-4112-b608-4cc64f56295c/kuadrant-console-plugin/0.log"
Nov 02 18:34:53.107242 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:53.107212    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/kuadrant-system_kuadrant-operator-controller-manager-678d87766f-47dcf_39b9a295-67cb-402a-9f11-41d0e8e7d2eb/manager/0.log"
Nov 02 18:34:55.031424 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.031384    4456 kubelet.go:2608] "SyncLoop ADD" source="api" pods=["openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"]
Nov 02 18:34:55.031845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.031788    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d4e4289f-eff9-4caa-9b01-7e0bc629a540" containerName="cleanup"
Nov 02 18:34:55.031845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.031799    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e4289f-eff9-4caa-9b01-7e0bc629a540" containerName="cleanup"
Nov 02 18:34:55.031845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.031836    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d4e4289f-eff9-4caa-9b01-7e0bc629a540" containerName="cleanup"
Nov 02 18:34:55.031845 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.031844    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e4289f-eff9-4caa-9b01-7e0bc629a540" containerName="cleanup"
Nov 02 18:34:55.031983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.031862    4456 cpu_manager.go:414] "containerMap: removing container" podUID="d4e4289f-eff9-4caa-9b01-7e0bc629a540" containerName="cleanup"
Nov 02 18:34:55.031983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.031871    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="d4e4289f-eff9-4caa-9b01-7e0bc629a540" containerName="cleanup"
Nov 02 18:34:55.031983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.031903    4456 cpu_manager.go:414] "containerMap: removing container" podUID="464f0262-7fb0-4a29-939a-3019ab473a3a" containerName="istio-proxy"
Nov 02 18:34:55.031983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.031908    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="464f0262-7fb0-4a29-939a-3019ab473a3a" containerName="istio-proxy"
Nov 02 18:34:55.031983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.031916    4456 cpu_manager.go:414] "containerMap: removing container" podUID="464f0262-7fb0-4a29-939a-3019ab473a3a" containerName="istio-proxy"
Nov 02 18:34:55.031983 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.031920    4456 state_mem.go:113] "Deleted CPUSet assignment" logger="CPUManager state memory" podUID="464f0262-7fb0-4a29-939a-3019ab473a3a" containerName="istio-proxy"
Nov 02 18:34:55.032486 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.031991    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="464f0262-7fb0-4a29-939a-3019ab473a3a" containerName="istio-proxy"
Nov 02 18:34:55.032486 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.031999    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="464f0262-7fb0-4a29-939a-3019ab473a3a" containerName="istio-proxy"
Nov 02 18:34:55.032486 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.032009    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e4289f-eff9-4caa-9b01-7e0bc629a540" containerName="cleanup"
Nov 02 18:34:55.032486 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.032016    4456 memory_manager.go:365] "RemoveStaleState removing state" podUID="d4e4289f-eff9-4caa-9b01-7e0bc629a540" containerName="cleanup"
Nov 02 18:34:55.035313 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.035294    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.051772 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.051745    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dt9rz\"/\"kube-root-ca.crt\""
Nov 02 18:34:55.051917 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.051785    4456 reflector.go:446] "Caches populated" type="*v1.ConfigMap" reflector="object-\"openshift-must-gather-dt9rz\"/\"openshift-service-ca.crt\""
Nov 02 18:34:55.052347 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.052320    4456 reflector.go:446] "Caches populated" type="*v1.Secret" reflector="object-\"openshift-must-gather-dt9rz\"/\"default-dockercfg-rxf8r\""
Nov 02 18:34:55.054225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.054206    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"]
Nov 02 18:34:55.083742 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.083704    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d08405ed-5012-4ed2-9b36-575551bfc2fc-sys\") pod \"perf-node-gather-daemonset-slgzf\" (UID: \"d08405ed-5012-4ed2-9b36-575551bfc2fc\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.083923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.083755    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d08405ed-5012-4ed2-9b36-575551bfc2fc-proc\") pod \"perf-node-gather-daemonset-slgzf\" (UID: \"d08405ed-5012-4ed2-9b36-575551bfc2fc\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.083923 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.083892    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d08405ed-5012-4ed2-9b36-575551bfc2fc-podres\") pod \"perf-node-gather-daemonset-slgzf\" (UID: \"d08405ed-5012-4ed2-9b36-575551bfc2fc\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.084034 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.083947    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d08405ed-5012-4ed2-9b36-575551bfc2fc-lib-modules\") pod \"perf-node-gather-daemonset-slgzf\" (UID: \"d08405ed-5012-4ed2-9b36-575551bfc2fc\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.084102 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.084085    4456 reconciler_common.go:251] "operationExecutor.VerifyControllerAttachedVolume started for volume \"kube-api-access-tfl9n\" (UniqueName: \"kubernetes.io/projected/d08405ed-5012-4ed2-9b36-575551bfc2fc-kube-api-access-tfl9n\") pod \"perf-node-gather-daemonset-slgzf\" (UID: \"d08405ed-5012-4ed2-9b36-575551bfc2fc\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.185233 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.185200    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"kube-api-access-tfl9n\" (UniqueName: \"kubernetes.io/projected/d08405ed-5012-4ed2-9b36-575551bfc2fc-kube-api-access-tfl9n\") pod \"perf-node-gather-daemonset-slgzf\" (UID: \"d08405ed-5012-4ed2-9b36-575551bfc2fc\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.185372 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.185240    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d08405ed-5012-4ed2-9b36-575551bfc2fc-sys\") pod \"perf-node-gather-daemonset-slgzf\" (UID: \"d08405ed-5012-4ed2-9b36-575551bfc2fc\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.185372 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.185266    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d08405ed-5012-4ed2-9b36-575551bfc2fc-proc\") pod \"perf-node-gather-daemonset-slgzf\" (UID: \"d08405ed-5012-4ed2-9b36-575551bfc2fc\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.185372 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.185291    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d08405ed-5012-4ed2-9b36-575551bfc2fc-podres\") pod \"perf-node-gather-daemonset-slgzf\" (UID: \"d08405ed-5012-4ed2-9b36-575551bfc2fc\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.185372 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.185328    4456 reconciler_common.go:225] "operationExecutor.MountVolume started for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d08405ed-5012-4ed2-9b36-575551bfc2fc-lib-modules\") pod \"perf-node-gather-daemonset-slgzf\" (UID: \"d08405ed-5012-4ed2-9b36-575551bfc2fc\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.185372 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.185342    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"sys\" (UniqueName: \"kubernetes.io/host-path/d08405ed-5012-4ed2-9b36-575551bfc2fc-sys\") pod \"perf-node-gather-daemonset-slgzf\" (UID: \"d08405ed-5012-4ed2-9b36-575551bfc2fc\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.185547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.185353    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"proc\" (UniqueName: \"kubernetes.io/host-path/d08405ed-5012-4ed2-9b36-575551bfc2fc-proc\") pod \"perf-node-gather-daemonset-slgzf\" (UID: \"d08405ed-5012-4ed2-9b36-575551bfc2fc\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.185547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.185393    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"podres\" (UniqueName: \"kubernetes.io/host-path/d08405ed-5012-4ed2-9b36-575551bfc2fc-podres\") pod \"perf-node-gather-daemonset-slgzf\" (UID: \"d08405ed-5012-4ed2-9b36-575551bfc2fc\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.185547 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.185414    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"lib-modules\" (UniqueName: \"kubernetes.io/host-path/d08405ed-5012-4ed2-9b36-575551bfc2fc-lib-modules\") pod \"perf-node-gather-daemonset-slgzf\" (UID: \"d08405ed-5012-4ed2-9b36-575551bfc2fc\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.209670 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.209639    4456 operation_generator.go:615] "MountVolume.SetUp succeeded for volume \"kube-api-access-tfl9n\" (UniqueName: \"kubernetes.io/projected/d08405ed-5012-4ed2-9b36-575551bfc2fc-kube-api-access-tfl9n\") pod \"perf-node-gather-daemonset-slgzf\" (UID: \"d08405ed-5012-4ed2-9b36-575551bfc2fc\") " pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.348076 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.347998    4456 util.go:34] "No sandbox for pod can be found. Need to start a new one" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:55.646583 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:55.646557    4456 kubelet.go:2615] "SyncLoop UPDATE" source="api" pods=["openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"]
Nov 02 18:34:55.647961 ip-10-0-141-87 kubenswrapper[4456]: W1102 18:34:55.647927    4456 manager.go:1172] Failed to process watch event {EventType:0 Name:/kubepods.slice/kubepods-podd08405ed_5012_4ed2_9b36_575551bfc2fc.slice/crio-690088f387e64cb98377eb60bfb9e4f574d60936b8ce42c889d4832084bef224 WatchSource:0}: Error finding container 690088f387e64cb98377eb60bfb9e4f574d60936b8ce42c889d4832084bef224: Status 404 returned error can't find the container with id 690088f387e64cb98377eb60bfb9e4f574d60936b8ce42c889d4832084bef224
Nov 02 18:34:56.203410 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:56.203383    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d987f75-ntp98_18d2ae0a-bf65-4aee-8730-093e00b54a24/metrics-server/1.log"
Nov 02 18:34:56.204575 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:56.204556    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_metrics-server-7d987f75-ntp98_18d2ae0a-bf65-4aee-8730-093e00b54a24/metrics-server/0.log"
Nov 02 18:34:56.268911 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:56.268869    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf" event={"ID":"d08405ed-5012-4ed2-9b36-575551bfc2fc","Type":"ContainerStarted","Data":"b7c72a4e817215b03efc44ad8ff57dbe8e4a24ad9a6fb66b10cd1837eddbe434"}
Nov 02 18:34:56.269087 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:56.268930    4456 kubelet.go:2639] "SyncLoop (PLEG): event for pod" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf" event={"ID":"d08405ed-5012-4ed2-9b36-575551bfc2fc","Type":"ContainerStarted","Data":"690088f387e64cb98377eb60bfb9e4f574d60936b8ce42c889d4832084bef224"}
Nov 02 18:34:56.269087 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:56.268982    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="not ready" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:34:56.313717 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:56.313655    4456 pod_startup_latency_tracker.go:148] "Observed pod startup duration" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf" podStartSLOduration=1.313641576 podStartE2EDuration="1.313641576s" totalImagesPullingTime="0s" totalInitContainerRuntime="0s" isStatefulPod=true podCreationTimestamp="2026-11-02 18:34:55 +0000 UTC" imagePullSessionsCount=0 imagePullSessionsStartsCount=0 observedRunningTime="2026-11-02 18:34:56.311769335 +0000 UTC m=+3912.802832498" watchObservedRunningTime="2026-11-02 18:34:56.313641576 +0000 UTC m=+3912.804704738"
Nov 02 18:34:56.426523 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:56.426498    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r45c5_8c89d394-c878-4069-b3a9-cd0abc822fec/node-exporter/0.log"
Nov 02 18:34:56.540837 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:56.540745    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r45c5_8c89d394-c878-4069-b3a9-cd0abc822fec/kube-rbac-proxy/0.log"
Nov 02 18:34:56.661198 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:56.661172    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_node-exporter-r45c5_8c89d394-c878-4069-b3a9-cd0abc822fec/init-textfile/0.log"
Nov 02 18:34:57.521411 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:57.521385    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-m89vg_9257d59b-6aaf-423e-b7ed-2e64dbdfa535/kube-rbac-proxy-main/0.log"
Nov 02 18:34:57.648225 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:57.640805    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-m89vg_9257d59b-6aaf-423e-b7ed-2e64dbdfa535/kube-rbac-proxy-self/0.log"
Nov 02 18:34:57.759584 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:57.759549    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_openshift-state-metrics-7bd7489957-m89vg_9257d59b-6aaf-423e-b7ed-2e64dbdfa535/openshift-state-metrics/0.log"
Nov 02 18:34:57.892840 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:57.892753    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/prometheus/0.log"
Nov 02 18:34:58.009663 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:58.009621    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/config-reloader/0.log"
Nov 02 18:34:58.127745 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:58.127717    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/thanos-sidecar/0.log"
Nov 02 18:34:58.246863 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:58.246838    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/kube-rbac-proxy-web/0.log"
Nov 02 18:34:58.367572 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:58.367538    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/kube-rbac-proxy/0.log"
Nov 02 18:34:58.486747 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:58.486715    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/kube-rbac-proxy-thanos/0.log"
Nov 02 18:34:58.606296 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:58.606214    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-k8s-0_83deda5b-6ef6-4a66-a76c-888303e6e834/init-config-reloader/0.log"
Nov 02 18:34:58.982614 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:58.982588    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-q2mbj_cae854c8-88f7-4201-b5d7-4e7c59fdf833/prometheus-operator-admission-webhook/2.log"
Nov 02 18:34:58.983305 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:58.983291    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_prometheus-operator-admission-webhook-6489d8bf89-q2mbj_cae854c8-88f7-4201-b5d7-4e7c59fdf833/prometheus-operator-admission-webhook/1.log"
Nov 02 18:34:59.080658 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:59.080623    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79c994cf5c-ckj5h_e065a5c3-95fd-4ac8-a287-8e1d20219201/telemeter-client/0.log"
Nov 02 18:34:59.196802 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:59.196775    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79c994cf5c-ckj5h_e065a5c3-95fd-4ac8-a287-8e1d20219201/reload/0.log"
Nov 02 18:34:59.313355 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:59.313278    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_telemeter-client-79c994cf5c-ckj5h_e065a5c3-95fd-4ac8-a287-8e1d20219201/kube-rbac-proxy/0.log"
Nov 02 18:34:59.437868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:59.437839    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/thanos-query/0.log"
Nov 02 18:34:59.556175 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:59.556150    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy-web/1.log"
Nov 02 18:34:59.556786 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:59.556763    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy-web/0.log"
Nov 02 18:34:59.644766 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:59.644695    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy/0.log"
Nov 02 18:34:59.768044 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:59.768019    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/prom-label-proxy/0.log"
Nov 02 18:34:59.885966 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:34:59.885937    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy-rules/0.log"
Nov 02 18:35:00.005548 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:35:00.005519    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-monitoring_thanos-querier-667db95b96-kmwrg_fb44e7b7-e0cb-4270-b14c-9c2397f37269/kube-rbac-proxy-metrics/0.log"
Nov 02 18:35:02.285571 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:35:02.285544    4456 kubelet.go:2727] "SyncLoop (probe)" probe="readiness" status="ready" pod="openshift-must-gather-dt9rz/perf-node-gather-daemonset-slgzf"
Nov 02 18:35:07.583599 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:35:07.583568    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zmrdt_f59eba5d-a500-462e-9ecd-4ef9e553f873/dns/0.log"
Nov 02 18:35:07.696262 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:35:07.696237    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_dns-default-zmrdt_f59eba5d-a500-462e-9ecd-4ef9e553f873/kube-rbac-proxy/0.log"
Nov 02 18:35:07.944364 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:35:07.944342    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-dns_node-resolver-hhsrb_ce8b7505-62a5-42f1-b432-d77200c21d2d/dns-node-resolver/0.log"
Nov 02 18:35:08.411025 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:35:08.410947    4456 log.go:25] "Finished parsing log file, hit bytes limit" path="/var/log/pods/kube-system_konnectivity-agent-mxsjq_367d8d85-a5e8-462d-b1d7-91a01e62b726/konnectivity-agent/0.log" limit=1024
Nov 02 18:35:09.199060 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:35:09.199027    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-59767d8795-7vwkt_0db36d92-4712-4338-958a-ebabf2c137ba/registry/1.log"
Nov 02 18:35:09.202516 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:35:09.202497    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_image-registry-59767d8795-7vwkt_0db36d92-4712-4338-958a-ebabf2c137ba/registry/0.log"
Nov 02 18:35:09.570261 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:35:09.570186    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-image-registry_node-ca-lhwgm_00b87b36-9ce2-480a-9924-22b33256fe1e/node-ca/0.log"
Nov 02 18:35:13.068286 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:35:13.068216    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_kube-auth-proxy-78648788bf-npsk7_347c8613-0791-494e-83e3-faded5593d88/kube-auth-proxy/0.log"
Nov 02 18:35:13.311026 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:35:13.310997    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-5knvx_8ef79a10-292b-42d9-aab0-53bee3e43ed2/istio-proxy/0.log"
Nov 02 18:35:13.655868 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:35:13.655838    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-lrssd_9d114747-c309-445d-8159-5e532bede175/istio-proxy/0.log"
Nov 02 18:35:13.720679 ip-10-0-141-87 kubenswrapper[4456]: I1102 18:35:13.720654    4456 log.go:25] "Finished parsing log file" path="/var/log/pods/openshift-ingress_maas-default-gateway-openshift-default-76f4876667-r5jnj_cce08d5b-d93c-4c5f-bb03-89c460372a06/istio-proxy/0.log"